Jul 28 00:44:37.285874 ip-10-0-137-173 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 00:44:37.285884 ip-10-0-137-173 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 00:44:37.285891 ip-10-0-137-173 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 00:44:37.286098 ip-10-0-137-173 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 00:44:47.471523 ip-10-0-137-173 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 00:44:47.471538 ip-10-0-137-173 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4461c2428bf446019f4a121c919a0608 -- Jul 28 00:46:44.128260 ip-10-0-137-173 systemd[1]: Starting Kubernetes Kubelet... Jul 28 00:46:44.507256 ip-10-0-137-173 kubenswrapper[2550]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 00:46:44.507256 ip-10-0-137-173 kubenswrapper[2550]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 00:46:44.507256 ip-10-0-137-173 kubenswrapper[2550]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 00:46:44.507256 ip-10-0-137-173 kubenswrapper[2550]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 00:46:44.507256 ip-10-0-137-173 kubenswrapper[2550]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 00:46:44.508843 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.508778 2550 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 00:46:44.513795 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513781 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513797 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513801 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513805 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513808 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513810 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513813 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513816 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513819 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513822 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513830 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513833 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:46:44.513832 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513836 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513839 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513842 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513845 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513847 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513850 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513852 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513855 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513857 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513860 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513862 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513865 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513876 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513879 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513881 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513885 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513887 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513889 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513892 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513894 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513897 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:46:44.514144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513899 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513902 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513904 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513907 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513909 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513912 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513914 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513918 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513922 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513925 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513928 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513931 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513934 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513938 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513941 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513944 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513946 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513949 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513952 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:46:44.514655 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513956 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513959 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513962 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513964 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513967 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513970 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513973 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513975 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513978 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513981 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513983 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513986 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513988 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513991 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513993 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513996 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.513998 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514001 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514014 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514017 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:46:44.515145 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514020 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514023 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514025 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514028 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514031 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514034 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514037 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514040 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514042 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514045 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514048 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514050 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514053 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514055 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514058 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514060 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514063 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514065 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514068 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514070 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:46:44.515611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514073 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514077 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514079 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514082 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514084 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514087 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514089 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514783 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514790 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514793 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514796 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514799 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514801 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514804 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514807 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514810 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514813 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514816 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514819 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514821 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:46:44.516111 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514824 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514827 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514829 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514832 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514834 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514837 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514840 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514843 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514845 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514847 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514851 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514853 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514856 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514859 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514864 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514867 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514870 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514873 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514876 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:46:44.516650 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514879 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514882 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514884 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514886 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514889 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514892 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514894 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514897 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514899 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514902 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514905 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514908 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514910 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514913 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514915 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514918 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514922 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514925 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514928 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514930 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:46:44.517159 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514933 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514935 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514938 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514940 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514943 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514945 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514948 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514950 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514953 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514955 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514959 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514961 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514964 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514966 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514969 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514971 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514974 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514976 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514980 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:46:44.517652 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514983 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514986 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514989 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514992 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514995 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.514998 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515001 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515017 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515020 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515024 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515026 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515029 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515032 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515035 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515038 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515041 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515044 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515047 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515049 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515052 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:46:44.518140 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515054 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515057 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515060 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515062 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515065 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515067 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515069 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515072 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515415 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515420 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515422 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515425 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515427 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515430 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515433 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515436 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515439 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515441 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515444 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515447 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:46:44.518653 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515449 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515452 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515455 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515457 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515461 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515465 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515468 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515471 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515474 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515476 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515479 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515482 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515485 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515487 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515490 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515493 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515495 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515498 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515501 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:46:44.519144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515504 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515506 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515509 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515511 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515514 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515516 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515519 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515522 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515525 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515527 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515530 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515532 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515535 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515537 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515540 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515542 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515544 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515547 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515549 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515551 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515554 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:46:44.519599 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515556 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515559 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515561 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515564 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515566 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515569 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515571 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515574 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515576 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515578 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515581 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515583 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515586 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515588 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515590 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515594 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515596 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515599 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515603 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515606 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:46:44.520107 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515608 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515611 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515613 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515616 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515619 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515621 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515624 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515627 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515629 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515632 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515634 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515637 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515641 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515644 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515647 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515649 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515654 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515657 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515660 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:46:44.520589 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515663 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515666 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515668 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515671 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515673 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515676 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515678 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.515681 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515746 2550 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515752 2550 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515758 2550 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515767 2550 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515771 2550 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515774 2550 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515778 2550 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515782 2550 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515785 2550 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515788 2550 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515791 2550 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515795 2550 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515797 2550 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515800 2550 flags.go:64] FLAG: --cgroup-root="" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515804 2550 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 00:46:44.521070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515807 2550 flags.go:64] FLAG: --client-ca-file="" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515810 2550 flags.go:64] FLAG: --cloud-provider="external" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515813 2550 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515817 2550 flags.go:64] FLAG: --cluster-domain="" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515819 2550 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515822 2550 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515827 2550 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515830 2550 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515834 2550 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515837 2550 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515840 2550 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515843 2550 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515846 2550 flags.go:64] FLAG: --contention-profiling="false" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515849 2550 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515852 2550 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515855 2550 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515858 2550 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515862 2550 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515865 2550 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515868 2550 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515872 2550 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515875 2550 flags.go:64] FLAG: --enable-server="true" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515877 2550 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515882 2550 flags.go:64] FLAG: --event-burst="100" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515885 2550 flags.go:64] FLAG: --event-qps="50" Jul 28 00:46:44.521607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515888 2550 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515891 2550 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515893 2550 flags.go:64] FLAG: --eviction-hard="" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515897 2550 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515900 2550 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515903 2550 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515906 2550 flags.go:64] FLAG: --eviction-soft="" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515909 2550 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515912 2550 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515914 2550 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515917 2550 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515920 2550 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515923 2550 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515927 2550 flags.go:64] FLAG: --feature-gates="" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515930 2550 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515933 2550 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515936 2550 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515939 2550 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515942 2550 flags.go:64] FLAG: --healthz-port="10248" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515946 2550 flags.go:64] FLAG: --help="false" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515949 2550 flags.go:64] FLAG: --hostname-override="ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515952 2550 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515955 2550 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 00:46:44.522268 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515958 2550 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515961 2550 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515965 2550 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515967 2550 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515971 2550 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515974 2550 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515977 2550 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515980 2550 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515983 2550 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515986 2550 flags.go:64] FLAG: --kube-reserved="" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515989 2550 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515992 2550 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515995 2550 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.515998 2550 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516000 2550 flags.go:64] FLAG: --lock-file="" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516016 2550 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516019 2550 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516022 2550 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516027 2550 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516030 2550 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516033 2550 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516036 2550 flags.go:64] FLAG: --logging-format="text" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516039 2550 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516043 2550 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 00:46:44.522876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516046 2550 flags.go:64] FLAG: --manifest-url="" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516048 2550 flags.go:64] FLAG: --manifest-url-header="" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516052 2550 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516055 2550 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516059 2550 flags.go:64] FLAG: --max-pods="110" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516063 2550 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516066 2550 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516069 2550 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516072 2550 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516075 2550 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516078 2550 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516080 2550 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516089 2550 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516092 2550 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516095 2550 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516098 2550 flags.go:64] FLAG: --pod-cidr="" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516101 2550 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516106 2550 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516109 2550 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516112 2550 flags.go:64] FLAG: --pods-per-core="0" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516115 2550 flags.go:64] FLAG: --port="10250" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516118 2550 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516121 2550 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02684347506c0fe18" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516125 2550 flags.go:64] FLAG: --qos-reserved="" Jul 28 00:46:44.523723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516128 2550 flags.go:64] FLAG: --read-only-port="10255" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516131 2550 flags.go:64] FLAG: --register-node="true" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516133 2550 flags.go:64] FLAG: --register-with-taints="" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516137 2550 flags.go:64] FLAG: --registry-burst="10" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516140 2550 flags.go:64] FLAG: --registry-qps="5" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516142 2550 flags.go:64] FLAG: --reserved-cpus="" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516146 2550 flags.go:64] FLAG: --reserved-memory="" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516151 2550 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516154 2550 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516157 2550 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516159 2550 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516162 2550 flags.go:64] FLAG: --runonce="false" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516165 2550 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516172 2550 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516175 2550 flags.go:64] FLAG: --seccomp-default="false" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516178 2550 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516181 2550 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516184 2550 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516187 2550 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516190 2550 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516194 2550 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516197 2550 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516200 2550 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516203 2550 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516206 2550 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 00:46:44.524418 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516209 2550 flags.go:64] FLAG: --system-cgroups="" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516211 2550 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516217 2550 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516219 2550 flags.go:64] FLAG: --tls-cert-file="" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516222 2550 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516226 2550 flags.go:64] FLAG: --tls-min-version="" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516229 2550 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516232 2550 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516235 2550 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516237 2550 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516240 2550 flags.go:64] FLAG: --v="2" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516244 2550 flags.go:64] FLAG: --version="false" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516248 2550 flags.go:64] FLAG: --vmodule="" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516251 2550 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.516256 2550 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516381 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516385 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516388 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516391 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516394 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516397 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516399 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516403 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:46:44.525060 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516405 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516408 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516411 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516413 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516417 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516420 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516422 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516425 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516428 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516430 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516433 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516436 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516438 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516441 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516444 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516446 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516449 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516451 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516454 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516456 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:46:44.525618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516459 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516461 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516465 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516470 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516473 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516476 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516478 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516481 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516484 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516487 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516489 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516492 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516495 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516498 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516500 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516503 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516507 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516509 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516512 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:46:44.526133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516515 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516517 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516520 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516523 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516525 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516528 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516530 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516533 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516535 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516538 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516541 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516543 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516546 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516548 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516551 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516553 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516559 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516563 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516566 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:46:44.526618 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516569 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516572 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516575 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516577 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516580 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516582 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516585 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516589 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516591 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516594 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516598 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516600 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516603 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516605 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516608 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516611 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516613 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516616 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516618 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:46:44.527087 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516622 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516625 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516628 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516630 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516633 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516636 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516638 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516641 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516643 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516646 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516649 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516652 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516655 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.516657 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.517333 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 00:46:44.527547 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.524859 2550 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.524872 2550 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.525661 2550 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.525669 2550 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525723 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525728 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525732 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525735 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525738 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525741 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525744 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525747 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525749 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525752 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525754 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525759 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525762 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525765 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:46:44.527917 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525768 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525771 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525774 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525777 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525779 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525782 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525784 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525787 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525790 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525792 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525797 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525801 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525804 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525807 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525810 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525813 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525815 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525818 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525821 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:46:44.528369 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525824 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525826 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525829 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525832 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525834 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525837 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525839 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525842 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525844 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525847 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525849 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525851 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525854 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525857 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525859 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525862 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525865 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525874 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525878 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525881 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:46:44.528839 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525884 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525887 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525889 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525892 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525895 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525897 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525900 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525903 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525905 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525907 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525910 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525913 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525915 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525918 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525920 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525922 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525925 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525927 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525930 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525932 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525935 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:46:44.529346 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525937 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525940 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525942 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525945 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525947 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525951 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525954 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525956 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525959 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525961 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525964 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525966 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525969 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525971 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525974 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525977 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525979 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525982 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525984 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525987 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:46:44.529896 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525989 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525992 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525995 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.525999 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526015 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.526021 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526126 2550 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526131 2550 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526135 2550 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526138 2550 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526141 2550 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526145 2550 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526148 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526150 2550 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526153 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 00:46:44.530583 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526156 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526158 2550 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526161 2550 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526164 2550 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526166 2550 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526169 2550 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526172 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526174 2550 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526177 2550 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526179 2550 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526182 2550 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526186 2550 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526189 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526193 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526196 2550 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526199 2550 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526202 2550 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526204 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526207 2550 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 00:46:44.530944 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526210 2550 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526213 2550 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526216 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526218 2550 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526221 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526224 2550 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526227 2550 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526230 2550 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526233 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526235 2550 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526238 2550 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526241 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526244 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526246 2550 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526249 2550 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526251 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526254 2550 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526257 2550 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526260 2550 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526263 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 00:46:44.531502 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526265 2550 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526268 2550 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526270 2550 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526273 2550 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526275 2550 feature_gate.go:328] unrecognized feature gate: Example Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526278 2550 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526281 2550 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526283 2550 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526285 2550 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526288 2550 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526291 2550 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526295 2550 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526298 2550 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526301 2550 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526305 2550 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526308 2550 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526310 2550 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526313 2550 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526315 2550 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526318 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 00:46:44.532121 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526320 2550 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526323 2550 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526325 2550 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526328 2550 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526330 2550 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526333 2550 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526335 2550 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526338 2550 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526340 2550 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526343 2550 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526346 2550 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526348 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526351 2550 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526353 2550 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526356 2550 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526358 2550 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526360 2550 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526363 2550 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526365 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 00:46:44.532915 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526368 2550 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526370 2550 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526373 2550 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526375 2550 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526378 2550 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526380 2550 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526383 2550 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526386 2550 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526390 2550 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526393 2550 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526396 2550 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:44.526399 2550 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.526404 2550 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.526497 2550 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 00:46:44.533390 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.528596 2550 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 00:46:44.533782 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.529957 2550 server.go:1019] "Starting client certificate rotation" Jul 28 00:46:44.533782 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.530058 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 00:46:44.533782 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.530096 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 00:46:44.533782 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.532875 2550 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 00:46:44.534082 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.534042 2550 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 00:46:44.542459 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.542446 2550 log.go:25] "Validated CRI v1 runtime API" Jul 28 00:46:44.545510 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.545495 2550 log.go:25] "Validated CRI v1 image API" Jul 28 00:46:44.546294 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.546281 2550 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 00:46:44.548721 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.548700 2550 fs.go:135] Filesystem UUIDs: map[47185165-ee19-43e7-a4f7-40796a64d8e9:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 877de922-67c3-4f0e-bd30-cfa78d1f7741:/dev/nvme0n1p3] Jul 28 00:46:44.548803 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.548720 2550 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 00:46:44.554180 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.554066 2550 manager.go:216] Machine: {Timestamp:2026-07-28 00:46:44.552786945 +0000 UTC m=+0.313634950 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099363 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bf5a76458e498446847051b08c97c SystemUUID:ec2bf5a7-6458-e498-4468-47051b08c97c BootID:4461c242-8bf4-4601-9f4a-121c919a0608 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a1:5e:a1:1b:95 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a1:5e:a1:1b:95 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:1b:a8:ba:32:15 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 00:46:44.554180 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.554167 2550 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 00:46:44.554307 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.554231 2550 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 00:46:44.554511 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.554491 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 00:46:44.556775 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.556748 2550 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 00:46:44.556895 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.556778 2550 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-173.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 00:46:44.557467 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.557457 2550 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 00:46:44.557504 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.557469 2550 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 00:46:44.557504 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.557476 2550 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 00:46:44.558080 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.558069 2550 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 00:46:44.559336 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.559326 2550 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 00:46:44.561965 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.561953 2550 state_mem.go:36] "Initialized new in-memory state store" Jul 28 00:46:44.562092 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.562082 2550 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 00:46:44.562131 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.562111 2550 kubelet.go:486] "Attempting to sync node with API server" Jul 28 00:46:44.562131 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.562119 2550 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 00:46:44.562131 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.562128 2550 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 00:46:44.562235 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.562137 2550 kubelet.go:398] "Adding apiserver pod source" Jul 28 00:46:44.562235 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.562145 2550 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 00:46:44.562469 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.562457 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 00:46:44.562519 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.562480 2550 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 00:46:44.566189 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.566168 2550 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 00:46:44.567466 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.567453 2550 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 00:46:44.567537 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.567482 2550 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 00:46:44.568138 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568127 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 00:46:44.568175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568142 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 00:46:44.568175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568149 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 00:46:44.568175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568155 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 00:46:44.568175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568163 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 00:46:44.568175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568169 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 00:46:44.568316 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568214 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 00:46:44.568316 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568221 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 00:46:44.568316 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568228 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 00:46:44.568316 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568233 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 00:46:44.568316 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568241 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 00:46:44.568316 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568249 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 00:46:44.569016 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568971 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 00:46:44.569016 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.568981 2550 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 00:46:44.571340 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.571317 2550 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-173.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 00:46:44.571340 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.571313 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 00:46:44.571455 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.571422 2550 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-173.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 00:46:44.571836 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.571823 2550 server.go:1268] "Started kubelet" Jul 28 00:46:44.571978 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.571908 2550 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 00:46:44.572260 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.572235 2550 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 00:46:44.572321 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.572277 2550 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 00:46:44.572555 ip-10-0-137-173 systemd[1]: Started Kubernetes Kubelet. Jul 28 00:46:44.573878 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.573852 2550 server.go:310] "Adding debug handlers to kubelet server" Jul 28 00:46:44.573987 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.573904 2550 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 00:46:44.575429 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.575411 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dqlj6" Jul 28 00:46:44.577573 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.576607 2550 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-173.ec2.internal.18c64d94fbab7f3e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-173.ec2.internal,UID:ip-10-0-137-173.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-173.ec2.internal,},FirstTimestamp:2026-07-28 00:46:44.571799358 +0000 UTC m=+0.332647363,LastTimestamp:2026-07-28 00:46:44.571799358 +0000 UTC m=+0.332647363,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-173.ec2.internal,}" Jul 28 00:46:44.578741 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.578714 2550 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 00:46:44.581311 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.581291 2550 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 00:46:44.581311 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.581307 2550 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 00:46:44.581725 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.581525 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:44.581994 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.581970 2550 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 00:46:44.582110 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.581971 2550 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 00:46:44.582110 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582056 2550 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 00:46:44.582212 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582178 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dqlj6" Jul 28 00:46:44.582263 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582187 2550 reconstruct.go:97] "Volume reconstruction finished" Jul 28 00:46:44.582263 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582235 2550 reconciler.go:29] "Reconciler: start to sync state" Jul 28 00:46:44.582355 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582341 2550 factory.go:153] Registering CRI-O factory Jul 28 00:46:44.582398 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582359 2550 factory.go:223] Registration of the crio container factory successfully Jul 28 00:46:44.582490 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582405 2550 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 00:46:44.582490 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582416 2550 factory.go:55] Registering systemd factory Jul 28 00:46:44.582490 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582426 2550 factory.go:223] Registration of the systemd container factory successfully Jul 28 00:46:44.582490 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582445 2550 factory.go:103] Registering Raw factory Jul 28 00:46:44.582490 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.582459 2550 manager.go:1196] Started watching for new ooms in manager Jul 28 00:46:44.583408 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.583384 2550 manager.go:349] Starting recovery of all containers Jul 28 00:46:44.591437 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.591417 2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:46:44.594395 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.594376 2550 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-173.ec2.internal\" not found" node="ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.595248 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.595235 2550 manager.go:354] Recovery completed Jul 28 00:46:44.598842 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.598831 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:46:44.600477 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.600463 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:46:44.600548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.600490 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:46:44.600548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.600504 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:46:44.601042 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.601031 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:46:44.601081 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.601070 2550 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 00:46:44.601126 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.601090 2550 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 00:46:44.601126 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.601109 2550 state_mem.go:36] "Initialized new in-memory state store" Jul 28 00:46:44.601609 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.601596 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:46:44.601657 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.601618 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:46:44.601657 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.601628 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:46:44.602158 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.602147 2550 policy_none.go:49] "None policy: Start" Jul 28 00:46:44.602190 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.602161 2550 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 00:46:44.602190 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.602171 2550 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 00:46:44.603162 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.603151 2550 policy_none.go:47] "Start" Jul 28 00:46:44.652623 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.652532 2550 manager.go:337] "Starting Device Plugin manager" Jul 28 00:46:44.652707 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.652666 2550 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 00:46:44.652707 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.652682 2550 server.go:87] "Starting device plugin registration server" Jul 28 00:46:44.652879 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.652869 2550 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 00:46:44.652912 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.652880 2550 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 00:46:44.652993 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.652971 2550 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 00:46:44.653114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.653071 2550 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 00:46:44.653114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.653081 2550 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 00:46:44.653467 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.653446 2550 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 00:46:44.653546 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.653497 2550 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:44.753201 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.753182 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:46:44.754255 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.753896 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:46:44.754255 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.753926 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:46:44.754255 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.753936 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:46:44.754255 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.753956 2550 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.761641 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.761608 2550 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.761641 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.761627 2550 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-173.ec2.internal\": node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:44.776558 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.776536 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 00:46:44.777575 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.777560 2550 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 00:46:44.777630 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.777592 2550 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 00:46:44.777630 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.777610 2550 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 00:46:44.777692 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.777639 2550 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 00:46:44.778270 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.778246 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:44.779864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.779849 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:46:44.878693 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.878667 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:44.878693 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.878683 2550 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal"] Jul 28 00:46:44.878821 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.878738 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:46:44.879538 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.879522 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:46:44.879612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.879553 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:46:44.879612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.879563 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:46:44.880832 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.880819 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:46:44.880948 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.880933 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.880995 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.880963 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:46:44.881533 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.881510 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:46:44.881533 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.881536 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:46:44.881657 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.881550 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:46:44.882188 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.882168 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:46:44.882291 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.882195 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:46:44.882291 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.882212 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:46:44.883417 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.883394 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.883496 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.883427 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33cddadf0b9611b00f916f9e5eb1693c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"33cddadf0b9611b00f916f9e5eb1693c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.883496 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.883443 2550 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 00:46:44.883496 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.883459 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33cddadf0b9611b00f916f9e5eb1693c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"33cddadf0b9611b00f916f9e5eb1693c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.883496 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.883486 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f1301d4be0daa7e23534036d6f77d33-config\") pod \"kube-apiserver-proxy-ip-10-0-137-173.ec2.internal\" (UID: \"4f1301d4be0daa7e23534036d6f77d33\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.886642 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.886624 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientMemory" Jul 28 00:46:44.886726 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.886656 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 00:46:44.886726 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.886672 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeHasSufficientPID" Jul 28 00:46:44.898644 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.898622 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-173.ec2.internal\" not found" node="ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.903906 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.903889 2550 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-173.ec2.internal\" not found" node="ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.979366 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:44.979342 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:44.983898 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.983883 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33cddadf0b9611b00f916f9e5eb1693c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"33cddadf0b9611b00f916f9e5eb1693c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.983960 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.983905 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33cddadf0b9611b00f916f9e5eb1693c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"33cddadf0b9611b00f916f9e5eb1693c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.983960 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.983924 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f1301d4be0daa7e23534036d6f77d33-config\") pod \"kube-apiserver-proxy-ip-10-0-137-173.ec2.internal\" (UID: \"4f1301d4be0daa7e23534036d6f77d33\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.984073 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.983980 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4f1301d4be0daa7e23534036d6f77d33-config\") pod \"kube-apiserver-proxy-ip-10-0-137-173.ec2.internal\" (UID: \"4f1301d4be0daa7e23534036d6f77d33\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.984073 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.984026 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33cddadf0b9611b00f916f9e5eb1693c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"33cddadf0b9611b00f916f9e5eb1693c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Jul 28 00:46:44.984073 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:44.984029 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/33cddadf0b9611b00f916f9e5eb1693c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal\" (UID: \"33cddadf0b9611b00f916f9e5eb1693c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Jul 28 00:46:45.080643 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:45.080585 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:45.188284 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:45.188245 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:45.201386 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.201360 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Jul 28 00:46:45.208157 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.208138 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Jul 28 00:46:45.289065 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:45.289035 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:45.390059 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:45.389979 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:45.490354 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:45.490320 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:45.530122 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.530090 2550 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 00:46:45.530584 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.530227 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 00:46:45.530584 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.530291 2550 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 00:46:45.588311 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.588279 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 00:41:44 +0000 UTC" deadline="2028-04-11 03:55:58.88676775 +0000 UTC" Jul 28 00:46:45.588311 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.588307 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14955h9m13.298463665s" Jul 28 00:46:45.590554 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:45.590535 2550 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-173.ec2.internal\" not found" Jul 28 00:46:45.658425 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.658360 2550 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:46:45.675879 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.675851 2550 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:46:45.681941 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.681914 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" Jul 28 00:46:45.694760 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.694737 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 00:46:45.695821 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.695806 2550 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" Jul 28 00:46:45.704103 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.704082 2550 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 00:46:45.852668 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:45.852640 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cddadf0b9611b00f916f9e5eb1693c.slice/crio-c09ecc871fd9013c4af53af86d4def397558f095f0840fb696b58c0487359f60 WatchSource:0}: Error finding container c09ecc871fd9013c4af53af86d4def397558f095f0840fb696b58c0487359f60: Status 404 returned error can't find the container with id c09ecc871fd9013c4af53af86d4def397558f095f0840fb696b58c0487359f60 Jul 28 00:46:45.852866 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:45.852848 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1301d4be0daa7e23534036d6f77d33.slice/crio-0f87162b0c5d594015c699247b84cec9d621e83db2ba466faa0b5707479fa862 WatchSource:0}: Error finding container 0f87162b0c5d594015c699247b84cec9d621e83db2ba466faa0b5707479fa862: Status 404 returned error can't find the container with id 0f87162b0c5d594015c699247b84cec9d621e83db2ba466faa0b5707479fa862 Jul 28 00:46:45.857758 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:45.857741 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:46:46.381308 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.381119 2550 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:46:46.563765 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.563739 2550 apiserver.go:52] "Watching apiserver" Jul 28 00:46:46.571632 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.570621 2550 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 00:46:46.571632 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.571160 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-svcsw","kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2","openshift-cluster-node-tuning-operator/tuned-bb8x9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal","openshift-multus/multus-6hd56","openshift-multus/multus-additional-cni-plugins-6sh29","openshift-multus/network-metrics-daemon-gddjn","kube-system/global-pull-secret-syncer-79x2c","openshift-dns/node-resolver-wwgrl","openshift-image-registry/node-ca-4dqfr","openshift-network-diagnostics/network-check-target-8jqzf","openshift-network-operator/iptables-alerter-45kvm","openshift-ovn-kubernetes/ovnkube-node-nmbmm"] Jul 28 00:46:46.574443 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.573378 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.578355 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.576603 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nskxd\"" Jul 28 00:46:46.578355 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.576907 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 00:46:46.578355 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.577163 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 00:46:46.578355 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.577175 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 00:46:46.578355 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.577385 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 00:46:46.578355 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.577398 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 00:46:46.578355 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.577545 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.581056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.578917 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.581056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.579949 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 00:46:46.581056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.580162 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bzjmc\"" Jul 28 00:46:46.581056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.580430 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 00:46:46.581056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.580608 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 00:46:46.581056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.580802 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.581888 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.581461 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5ct4w\"" Jul 28 00:46:46.581888 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.581758 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 00:46:46.584056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.582380 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 00:46:46.584056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.582485 2550 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 00:46:46.584232 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.584196 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 00:46:46.584378 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.584311 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.584623 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.584597 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 00:46:46.584816 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.584798 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fdn7s\"" Jul 28 00:46:46.585741 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.585719 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.586047 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.586026 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 00:46:46.587371 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.587351 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:46.587496 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.587454 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:46:46.588781 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.588749 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvglm\"" Jul 28 00:46:46.594923 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.593469 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 00:46:46.594923 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.593484 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vzxbh\"" Jul 28 00:46:46.594923 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.593857 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 00:46:46.594923 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.594044 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 00:46:46.594923 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.594171 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 00:46:46.594923 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.594420 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 00:46:46.594923 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.594614 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.594923 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.594641 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 00:46:46.595443 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595037 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595628 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bn4k\" (UniqueName: \"kubernetes.io/projected/eb91259a-5111-49b5-bf8d-f3d2404180c2-kube-api-access-7bn4k\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595665 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c8da112-99aa-4570-9c43-197db572fa95-host\") pod \"node-ca-4dqfr\" (UID: \"7c8da112-99aa-4570-9c43-197db572fa95\") " pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595720 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-host\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595752 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-node-log\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595771 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-run-ovn\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595785 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-system-cni-dir\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595789 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595801 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-var-lib-cni-bin\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595816 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-var-lib-cni-multus\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595833 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-hostroot\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595847 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-systemd-units\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.595848 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595861 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-lib-modules\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595877 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-cni-bin\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595892 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-cnibin\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595926 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-os-release\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.597250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595957 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.595990 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596020 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsk8c\" (UniqueName: \"kubernetes.io/projected/9c3f258e-cca5-450c-960a-53d3aad3953d-kube-api-access-nsk8c\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596053 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-var-lib-kubelet\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596094 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-etc-openvswitch\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596125 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-device-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596147 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-run-systemd\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596172 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596195 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb91259a-5111-49b5-bf8d-f3d2404180c2-ovnkube-script-lib\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596267 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-multus-socket-dir-parent\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596294 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7858a6fd-db8f-4492-b847-d19cec86528d-multus-daemon-config\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596319 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-systemd\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596385 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-sys\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596415 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02567703-a907-465b-a016-386dd7208f74-etc-tuned\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596441 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-cni-binary-copy\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596480 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.601673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596548 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-socket-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596600 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-kubelet\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596630 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-run-netns\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596679 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb91259a-5111-49b5-bf8d-f3d2404180c2-env-overrides\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596714 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-multus-conf-dir\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596763 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-sysconfig\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596817 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-multus-cni-dir\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596877 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-cnibin\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596937 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-etc-kubernetes\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.596968 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-registration-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.597015 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-sys-fs\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.597061 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-sysctl-d\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.597106 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb91259a-5111-49b5-bf8d-f3d2404180c2-ovn-node-metrics-cert\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.597159 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-run-netns\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.598627 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599145 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29gnq\" (UniqueName: \"kubernetes.io/projected/7c8da112-99aa-4570-9c43-197db572fa95-kube-api-access-29gnq\") pod \"node-ca-4dqfr\" (UID: \"7c8da112-99aa-4570-9c43-197db572fa95\") " pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599184 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8hc\" (UniqueName: \"kubernetes.io/projected/02567703-a907-465b-a016-386dd7208f74-kube-api-access-sj8hc\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.602636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599210 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-slash\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599233 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599256 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c8da112-99aa-4570-9c43-197db572fa95-serviceca\") pod \"node-ca-4dqfr\" (UID: \"7c8da112-99aa-4570-9c43-197db572fa95\") " pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599282 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-kubernetes\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599306 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-log-socket\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599334 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599361 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-run-k8s-cni-cncf-io\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599386 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02567703-a907-465b-a016-386dd7208f74-tmp\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599411 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-run-multus-certs\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599438 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngcmj\" (UniqueName: \"kubernetes.io/projected/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-kube-api-access-ngcmj\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599466 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-sysctl-conf\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599497 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqgbj\" (UniqueName: \"kubernetes.io/projected/7858a6fd-db8f-4492-b847-d19cec86528d-kube-api-access-nqgbj\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599541 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-system-cni-dir\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599566 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599593 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-var-lib-openvswitch\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599618 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-cni-netd\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.603650 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599668 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7858a6fd-db8f-4492-b847-d19cec86528d-cni-binary-copy\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599696 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-modprobe-d\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599743 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-run-openvswitch\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599793 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb91259a-5111-49b5-bf8d-f3d2404180c2-ovnkube-config\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599822 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-os-release\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599875 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-var-lib-kubelet\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.599920 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-run\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.600210 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4dgnx\"" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.603423 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.603651 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.604228 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 00:46:46.604487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.604427 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 00:46:46.605108 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.604609 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 00:46:46.605108 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.604792 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xt8b5\"" Jul 28 00:46:46.605108 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.605021 2550 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 00:46:46.606026 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.605456 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:46:46.607055 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.606759 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:46.607055 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.606820 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:46:46.608388 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.608189 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 00:46:46.608388 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.608247 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 00:46:46.608388 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.608306 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-29xrf\"" Jul 28 00:46:46.617712 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.617681 2550 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mzgd8" Jul 28 00:46:46.627029 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.626962 2550 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mzgd8" Jul 28 00:46:46.684061 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.684036 2550 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 00:46:46.701209 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701186 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-os-release\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.701335 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701225 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-var-lib-kubelet\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.701335 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701250 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-run\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.701448 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701327 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-var-lib-kubelet\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.701448 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701396 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:46.701448 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701420 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-os-release\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.701549 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701443 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bn4k\" (UniqueName: \"kubernetes.io/projected/eb91259a-5111-49b5-bf8d-f3d2404180c2-kube-api-access-7bn4k\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.701549 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701471 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c8da112-99aa-4570-9c43-197db572fa95-host\") pod \"node-ca-4dqfr\" (UID: \"7c8da112-99aa-4570-9c43-197db572fa95\") " pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.701549 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701495 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-host\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.701549 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701519 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-node-log\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.701549 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701544 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-run-ovn\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.701734 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701566 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-run\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.701734 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701567 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-system-cni-dir\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.701734 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701609 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-var-lib-cni-bin\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.701734 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701629 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-system-cni-dir\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.701734 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701648 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-var-lib-cni-multus\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.701734 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701674 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-hostroot\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.701734 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701700 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-systemd-units\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.701734 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701724 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-lib-modules\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701754 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4dfca9b2-d814-4d0f-8754-eec2a38a19fe-konnectivity-ca\") pod \"konnectivity-agent-svcsw\" (UID: \"4dfca9b2-d814-4d0f-8754-eec2a38a19fe\") " pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701780 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-dbus\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701812 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06-iptables-alerter-script\") pod \"iptables-alerter-45kvm\" (UID: \"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06\") " pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701839 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-cni-bin\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701863 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-cnibin\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701905 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-os-release\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701929 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c8da112-99aa-4570-9c43-197db572fa95-host\") pod \"node-ca-4dqfr\" (UID: \"7c8da112-99aa-4570-9c43-197db572fa95\") " pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701932 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701973 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701990 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.701996 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsk8c\" (UniqueName: \"kubernetes.io/projected/9c3f258e-cca5-450c-960a-53d3aad3953d-kube-api-access-nsk8c\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702024 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-var-lib-kubelet\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702051 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-etc-openvswitch\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702092 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-device-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.702143 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702125 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-kubelet-config\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702151 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmh88\" (UniqueName: \"kubernetes.io/projected/9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06-kube-api-access-zmh88\") pod \"iptables-alerter-45kvm\" (UID: \"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06\") " pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702192 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-run-systemd\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702214 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-host\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702219 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702250 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb91259a-5111-49b5-bf8d-f3d2404180c2-ovnkube-script-lib\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702268 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-multus-socket-dir-parent\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702270 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-var-lib-cni-bin\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702285 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7858a6fd-db8f-4492-b847-d19cec86528d-multus-daemon-config\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702301 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-systemd\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702316 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-sys\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702331 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02567703-a907-465b-a016-386dd7208f74-etc-tuned\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702344 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-hostroot\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702355 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-cni-binary-copy\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702380 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702382 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-systemd-units\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702398 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-socket-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.702864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702414 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-kubelet\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702429 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-run-netns\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702449 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb91259a-5111-49b5-bf8d-f3d2404180c2-env-overrides\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702475 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-multus-conf-dir\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702498 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-lib-modules\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702500 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-sysconfig\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702535 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-multus-cni-dir\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702539 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-sysconfig\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702565 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-cnibin\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702583 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-node-log\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702592 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-etc-kubernetes\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702611 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-run-ovn\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702618 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-registration-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702643 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-sys-fs\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702666 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-sysctl-d\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702681 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702695 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a38938d2-ef9d-43ec-91d9-575165164123-hosts-file\") pod \"node-resolver-wwgrl\" (UID: \"a38938d2-ef9d-43ec-91d9-575165164123\") " pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702707 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-run-ovn-kubernetes\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.703593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702721 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97n9b\" (UniqueName: \"kubernetes.io/projected/a38938d2-ef9d-43ec-91d9-575165164123-kube-api-access-97n9b\") pod \"node-resolver-wwgrl\" (UID: \"a38938d2-ef9d-43ec-91d9-575165164123\") " pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702749 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb91259a-5111-49b5-bf8d-f3d2404180c2-ovn-node-metrics-cert\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702775 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-run-netns\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702800 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29gnq\" (UniqueName: \"kubernetes.io/projected/7c8da112-99aa-4570-9c43-197db572fa95-kube-api-access-29gnq\") pod \"node-ca-4dqfr\" (UID: \"7c8da112-99aa-4570-9c43-197db572fa95\") " pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702829 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8hc\" (UniqueName: \"kubernetes.io/projected/02567703-a907-465b-a016-386dd7208f74-kube-api-access-sj8hc\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702853 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4dfca9b2-d814-4d0f-8754-eec2a38a19fe-agent-certs\") pod \"konnectivity-agent-svcsw\" (UID: \"4dfca9b2-d814-4d0f-8754-eec2a38a19fe\") " pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702880 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-slash\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702907 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702938 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c8da112-99aa-4570-9c43-197db572fa95-serviceca\") pod \"node-ca-4dqfr\" (UID: \"7c8da112-99aa-4570-9c43-197db572fa95\") " pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702964 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-kubernetes\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702991 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-log-socket\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703019 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703050 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-run-k8s-cni-cncf-io\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703075 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02567703-a907-465b-a016-386dd7208f74-tmp\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703118 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703145 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06-host-slash\") pod \"iptables-alerter-45kvm\" (UID: \"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06\") " pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703171 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:46.704556 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703198 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-run-multus-certs\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703227 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngcmj\" (UniqueName: \"kubernetes.io/projected/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-kube-api-access-ngcmj\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703254 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-sysctl-conf\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703274 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eb91259a-5111-49b5-bf8d-f3d2404180c2-ovnkube-script-lib\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703282 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztgsk\" (UniqueName: \"kubernetes.io/projected/07a0f217-710e-482b-a1bb-a358656d435f-kube-api-access-ztgsk\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703311 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqgbj\" (UniqueName: \"kubernetes.io/projected/7858a6fd-db8f-4492-b847-d19cec86528d-kube-api-access-nqgbj\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-system-cni-dir\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703358 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703387 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-var-lib-openvswitch\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703412 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-cni-netd\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703442 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7858a6fd-db8f-4492-b847-d19cec86528d-cni-binary-copy\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703471 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-modprobe-d\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703497 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a38938d2-ef9d-43ec-91d9-575165164123-tmp-dir\") pod \"node-resolver-wwgrl\" (UID: \"a38938d2-ef9d-43ec-91d9-575165164123\") " pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703533 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-run-openvswitch\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703593 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb91259a-5111-49b5-bf8d-f3d2404180c2-ovnkube-config\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703692 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-slash\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703797 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-sysctl-conf\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.705422 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703871 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7858a6fd-db8f-4492-b847-d19cec86528d-multus-daemon-config\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703941 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-systemd\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704066 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704094 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-system-cni-dir\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704112 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-socket-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704141 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-run-netns\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.702313 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-var-lib-cni-multus\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704148 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-kubelet\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.703359 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-multus-socket-dir-parent\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704188 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-sys\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704188 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eb91259a-5111-49b5-bf8d-f3d2404180c2-ovnkube-config\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704202 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-var-lib-openvswitch\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704294 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704420 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-cni-bin\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704472 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-cnibin\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704535 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-os-release\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704572 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eb91259a-5111-49b5-bf8d-f3d2404180c2-env-overrides\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.706302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704580 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-var-lib-kubelet\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704614 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-multus-conf-dir\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704642 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-cni-netd\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704648 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-etc-openvswitch\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704680 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704701 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-kubernetes\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704702 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-device-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704737 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7c8da112-99aa-4570-9c43-197db572fa95-serviceca\") pod \"node-ca-4dqfr\" (UID: \"7c8da112-99aa-4570-9c43-197db572fa95\") " pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704746 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-log-socket\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704741 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7858a6fd-db8f-4492-b847-d19cec86528d-cni-binary-copy\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704842 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-modprobe-d\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704896 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-run-openvswitch\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704920 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-run-systemd\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704943 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-cnibin\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704952 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-run-multus-certs\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704959 2550 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704987 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-sys-fs\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704999 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c3f258e-cca5-450c-960a-53d3aad3953d-registration-dir\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.707217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.705007 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-run-netns\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.708156 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.705035 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-etc-kubernetes\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.708156 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.705069 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/02567703-a907-465b-a016-386dd7208f74-etc-sysctl-d\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.708156 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.705124 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb91259a-5111-49b5-bf8d-f3d2404180c2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.708156 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.704963 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-host-run-k8s-cni-cncf-io\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.708156 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.705367 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7858a6fd-db8f-4492-b847-d19cec86528d-multus-cni-dir\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.708156 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.705521 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-cni-binary-copy\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.708884 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.708857 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/02567703-a907-465b-a016-386dd7208f74-tmp\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.708977 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.708893 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb91259a-5111-49b5-bf8d-f3d2404180c2-ovn-node-metrics-cert\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.710074 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.710055 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/02567703-a907-465b-a016-386dd7208f74-etc-tuned\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.716934 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.716875 2550 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 00:46:46.717499 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.717478 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqgbj\" (UniqueName: \"kubernetes.io/projected/7858a6fd-db8f-4492-b847-d19cec86528d-kube-api-access-nqgbj\") pod \"multus-6hd56\" (UID: \"7858a6fd-db8f-4492-b847-d19cec86528d\") " pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.718363 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.718316 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bn4k\" (UniqueName: \"kubernetes.io/projected/eb91259a-5111-49b5-bf8d-f3d2404180c2-kube-api-access-7bn4k\") pod \"ovnkube-node-nmbmm\" (UID: \"eb91259a-5111-49b5-bf8d-f3d2404180c2\") " pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.720775 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.720749 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngcmj\" (UniqueName: \"kubernetes.io/projected/a14ecd2d-fc55-40dd-85fa-6152eb2ce28c-kube-api-access-ngcmj\") pod \"multus-additional-cni-plugins-6sh29\" (UID: \"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c\") " pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.721609 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.721568 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8hc\" (UniqueName: \"kubernetes.io/projected/02567703-a907-465b-a016-386dd7208f74-kube-api-access-sj8hc\") pod \"tuned-bb8x9\" (UID: \"02567703-a907-465b-a016-386dd7208f74\") " pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.722025 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.721991 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsk8c\" (UniqueName: \"kubernetes.io/projected/9c3f258e-cca5-450c-960a-53d3aad3953d-kube-api-access-nsk8c\") pod \"aws-ebs-csi-driver-node-vfrg2\" (UID: \"9c3f258e-cca5-450c-960a-53d3aad3953d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.722459 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.722437 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29gnq\" (UniqueName: \"kubernetes.io/projected/7c8da112-99aa-4570-9c43-197db572fa95-kube-api-access-29gnq\") pod \"node-ca-4dqfr\" (UID: \"7c8da112-99aa-4570-9c43-197db572fa95\") " pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.783096 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.782997 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" event={"ID":"4f1301d4be0daa7e23534036d6f77d33","Type":"ContainerStarted","Data":"0f87162b0c5d594015c699247b84cec9d621e83db2ba466faa0b5707479fa862"} Jul 28 00:46:46.784609 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.784582 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" event={"ID":"33cddadf0b9611b00f916f9e5eb1693c","Type":"ContainerStarted","Data":"c09ecc871fd9013c4af53af86d4def397558f095f0840fb696b58c0487359f60"} Jul 28 00:46:46.803941 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.803914 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-kubelet-config\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:46.804149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.803954 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmh88\" (UniqueName: \"kubernetes.io/projected/9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06-kube-api-access-zmh88\") pod \"iptables-alerter-45kvm\" (UID: \"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06\") " pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.804149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804064 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-kubelet-config\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:46.804149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804098 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a38938d2-ef9d-43ec-91d9-575165164123-hosts-file\") pod \"node-resolver-wwgrl\" (UID: \"a38938d2-ef9d-43ec-91d9-575165164123\") " pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.804149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804139 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97n9b\" (UniqueName: \"kubernetes.io/projected/a38938d2-ef9d-43ec-91d9-575165164123-kube-api-access-97n9b\") pod \"node-resolver-wwgrl\" (UID: \"a38938d2-ef9d-43ec-91d9-575165164123\") " pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.804325 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804173 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4dfca9b2-d814-4d0f-8754-eec2a38a19fe-agent-certs\") pod \"konnectivity-agent-svcsw\" (UID: \"4dfca9b2-d814-4d0f-8754-eec2a38a19fe\") " pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:46:46.804325 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804206 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:46.804325 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804216 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a38938d2-ef9d-43ec-91d9-575165164123-hosts-file\") pod \"node-resolver-wwgrl\" (UID: \"a38938d2-ef9d-43ec-91d9-575165164123\") " pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.804325 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804232 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06-host-slash\") pod \"iptables-alerter-45kvm\" (UID: \"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06\") " pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.804325 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804257 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:46.804325 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804287 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztgsk\" (UniqueName: \"kubernetes.io/projected/07a0f217-710e-482b-a1bb-a358656d435f-kube-api-access-ztgsk\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:46.804325 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804317 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a38938d2-ef9d-43ec-91d9-575165164123-tmp-dir\") pod \"node-resolver-wwgrl\" (UID: \"a38938d2-ef9d-43ec-91d9-575165164123\") " pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.804585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804348 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:46.804585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804372 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06-host-slash\") pod \"iptables-alerter-45kvm\" (UID: \"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06\") " pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.804585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804385 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4dfca9b2-d814-4d0f-8754-eec2a38a19fe-konnectivity-ca\") pod \"konnectivity-agent-svcsw\" (UID: \"4dfca9b2-d814-4d0f-8754-eec2a38a19fe\") " pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:46:46.804585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804416 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-dbus\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:46.804585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804441 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06-iptables-alerter-script\") pod \"iptables-alerter-45kvm\" (UID: \"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06\") " pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.804585 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.804480 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:46.804585 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.804569 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret podName:a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.304526156 +0000 UTC m=+3.065374171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret") pod "global-pull-secret-syncer-79x2c" (UID: "a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:46.804934 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804677 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a38938d2-ef9d-43ec-91d9-575165164123-tmp-dir\") pod \"node-resolver-wwgrl\" (UID: \"a38938d2-ef9d-43ec-91d9-575165164123\") " pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.804934 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.804797 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:46.804934 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.804843 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-dbus\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:46.804934 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.804874 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs podName:07a0f217-710e-482b-a1bb-a358656d435f nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.304855664 +0000 UTC m=+3.065703664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs") pod "network-metrics-daemon-gddjn" (UID: "07a0f217-710e-482b-a1bb-a358656d435f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:46.805134 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.805113 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06-iptables-alerter-script\") pod \"iptables-alerter-45kvm\" (UID: \"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06\") " pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.805275 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.805254 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4dfca9b2-d814-4d0f-8754-eec2a38a19fe-konnectivity-ca\") pod \"konnectivity-agent-svcsw\" (UID: \"4dfca9b2-d814-4d0f-8754-eec2a38a19fe\") " pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:46:46.807685 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.807662 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4dfca9b2-d814-4d0f-8754-eec2a38a19fe-agent-certs\") pod \"konnectivity-agent-svcsw\" (UID: \"4dfca9b2-d814-4d0f-8754-eec2a38a19fe\") " pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:46:46.812705 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.812675 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:46:46.812705 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.812701 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:46:46.812851 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.812716 2550 projected.go:196] Error preparing data for projected volume kube-api-access-nbjvp for pod openshift-network-diagnostics/network-check-target-8jqzf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:46.812851 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:46.812774 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp podName:57227b22-d127-4005-90b6-cc2d78d2fd38 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:47.312759045 +0000 UTC m=+3.073607040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjvp" (UniqueName: "kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp") pod "network-check-target-8jqzf" (UID: "57227b22-d127-4005-90b6-cc2d78d2fd38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:46.816027 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.816002 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97n9b\" (UniqueName: \"kubernetes.io/projected/a38938d2-ef9d-43ec-91d9-575165164123-kube-api-access-97n9b\") pod \"node-resolver-wwgrl\" (UID: \"a38938d2-ef9d-43ec-91d9-575165164123\") " pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.816463 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.816419 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmh88\" (UniqueName: \"kubernetes.io/projected/9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06-kube-api-access-zmh88\") pod \"iptables-alerter-45kvm\" (UID: \"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06\") " pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.816570 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.816469 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztgsk\" (UniqueName: \"kubernetes.io/projected/07a0f217-710e-482b-a1bb-a358656d435f-kube-api-access-ztgsk\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:46.901606 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.901537 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6sh29" Jul 28 00:46:46.917755 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.917727 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4dqfr" Jul 28 00:46:46.927962 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.927920 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" Jul 28 00:46:46.934982 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.934962 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" Jul 28 00:46:46.942775 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.942755 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:46:46.949532 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.949513 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6hd56" Jul 28 00:46:46.958255 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.958237 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-45kvm" Jul 28 00:46:46.966945 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.966924 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wwgrl" Jul 28 00:46:46.977739 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:46.977720 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:46:47.309740 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:47.309643 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:47.309740 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:47.309725 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:47.309947 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:47.309928 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:47.310027 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:47.310014 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret podName:a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:48.309993538 +0000 UTC m=+4.070841555 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret") pod "global-pull-secret-syncer-79x2c" (UID: "a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:47.310628 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:47.310607 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:47.310709 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:47.310678 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs podName:07a0f217-710e-482b-a1bb-a358656d435f nodeName:}" failed. No retries permitted until 2026-07-28 00:46:48.310659587 +0000 UTC m=+4.071507580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs") pod "network-metrics-daemon-gddjn" (UID: "07a0f217-710e-482b-a1bb-a358656d435f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:47.411018 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:47.410987 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:47.411153 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:47.411134 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:46:47.411153 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:47.411149 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:46:47.411245 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:47.411160 2550 projected.go:196] Error preparing data for projected volume kube-api-access-nbjvp for pod openshift-network-diagnostics/network-check-target-8jqzf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:47.411245 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:47.411206 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp podName:57227b22-d127-4005-90b6-cc2d78d2fd38 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:48.41119354 +0000 UTC m=+4.172041533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjvp" (UniqueName: "kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp") pod "network-check-target-8jqzf" (UID: "57227b22-d127-4005-90b6-cc2d78d2fd38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:47.629538 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:47.629422 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 00:41:46 +0000 UTC" deadline="2028-01-18 10:33:42.426638676 +0000 UTC" Jul 28 00:46:47.629538 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:47.629480 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12945h46m54.797162225s" Jul 28 00:46:47.778405 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:47.778378 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:47.778575 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:47.778494 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:46:47.876181 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:47.876132 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c8da112_99aa_4570_9c43_197db572fa95.slice/crio-7fd66da9b9861bf7cea0c8a4109a7b78bc94a36037a54513124aca8b8dfc8654 WatchSource:0}: Error finding container 7fd66da9b9861bf7cea0c8a4109a7b78bc94a36037a54513124aca8b8dfc8654: Status 404 returned error can't find the container with id 7fd66da9b9861bf7cea0c8a4109a7b78bc94a36037a54513124aca8b8dfc8654 Jul 28 00:46:47.877870 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:47.877843 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dfca9b2_d814_4d0f_8754_eec2a38a19fe.slice/crio-b30d21edec7b66e3585560e58c9e4457ad0b6e21aa05c95d4ef41c9f7117fe33 WatchSource:0}: Error finding container b30d21edec7b66e3585560e58c9e4457ad0b6e21aa05c95d4ef41c9f7117fe33: Status 404 returned error can't find the container with id b30d21edec7b66e3585560e58c9e4457ad0b6e21aa05c95d4ef41c9f7117fe33 Jul 28 00:46:47.881422 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:47.881399 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02567703_a907_465b_a016_386dd7208f74.slice/crio-4ddaad87a0ee45e0f7de3ae80d5d4cc495fc04883dc33d4c554685597caa013f WatchSource:0}: Error finding container 4ddaad87a0ee45e0f7de3ae80d5d4cc495fc04883dc33d4c554685597caa013f: Status 404 returned error can't find the container with id 4ddaad87a0ee45e0f7de3ae80d5d4cc495fc04883dc33d4c554685597caa013f Jul 28 00:46:47.882556 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:47.882345 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c3f258e_cca5_450c_960a_53d3aad3953d.slice/crio-5db3e18ee832baa16a5aa9172f9419dcf719c89ce31444e4d707f9cd01850fa3 WatchSource:0}: Error finding container 5db3e18ee832baa16a5aa9172f9419dcf719c89ce31444e4d707f9cd01850fa3: Status 404 returned error can't find the container with id 5db3e18ee832baa16a5aa9172f9419dcf719c89ce31444e4d707f9cd01850fa3 Jul 28 00:46:47.883186 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:47.883157 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda14ecd2d_fc55_40dd_85fa_6152eb2ce28c.slice/crio-80fc382ce58581737b7ee3558183119a8a9bcb2887325179fc9983fc3e33d94a WatchSource:0}: Error finding container 80fc382ce58581737b7ee3558183119a8a9bcb2887325179fc9983fc3e33d94a: Status 404 returned error can't find the container with id 80fc382ce58581737b7ee3558183119a8a9bcb2887325179fc9983fc3e33d94a Jul 28 00:46:47.884459 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:46:47.884432 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb91259a_5111_49b5_bf8d_f3d2404180c2.slice/crio-553e493fa0b81609ad1c7a7fcdabff59b43ecc690f5a95a4e5418b96af9653f0 WatchSource:0}: Error finding container 553e493fa0b81609ad1c7a7fcdabff59b43ecc690f5a95a4e5418b96af9653f0: Status 404 returned error can't find the container with id 553e493fa0b81609ad1c7a7fcdabff59b43ecc690f5a95a4e5418b96af9653f0 Jul 28 00:46:48.317632 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.317345 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:48.317632 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.317599 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:48.317829 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.317471 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:48.317829 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.317699 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:48.317829 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.317705 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs podName:07a0f217-710e-482b-a1bb-a358656d435f nodeName:}" failed. No retries permitted until 2026-07-28 00:46:50.317679205 +0000 UTC m=+6.078527216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs") pod "network-metrics-daemon-gddjn" (UID: "07a0f217-710e-482b-a1bb-a358656d435f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:48.317829 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.317763 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret podName:a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:50.317746759 +0000 UTC m=+6.078594760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret") pod "global-pull-secret-syncer-79x2c" (UID: "a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:48.418475 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.418441 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:48.418660 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.418623 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:46:48.418660 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.418644 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:46:48.418660 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.418658 2550 projected.go:196] Error preparing data for projected volume kube-api-access-nbjvp for pod openshift-network-diagnostics/network-check-target-8jqzf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:48.418786 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.418715 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp podName:57227b22-d127-4005-90b6-cc2d78d2fd38 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:50.418695453 +0000 UTC m=+6.179543450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjvp" (UniqueName: "kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp") pod "network-check-target-8jqzf" (UID: "57227b22-d127-4005-90b6-cc2d78d2fd38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:48.630258 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.630142 2550 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 00:41:46 +0000 UTC" deadline="2028-03-07 05:57:46.025292563 +0000 UTC" Jul 28 00:46:48.630258 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.630181 2550 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14117h10m57.395116066s" Jul 28 00:46:48.780842 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.780806 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:48.781030 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.780953 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:46:48.781122 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.781037 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:48.781346 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:48.781132 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:46:48.809538 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.809497 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6sh29" event={"ID":"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c","Type":"ContainerStarted","Data":"80fc382ce58581737b7ee3558183119a8a9bcb2887325179fc9983fc3e33d94a"} Jul 28 00:46:48.825238 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.825185 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-svcsw" event={"ID":"4dfca9b2-d814-4d0f-8754-eec2a38a19fe","Type":"ContainerStarted","Data":"b30d21edec7b66e3585560e58c9e4457ad0b6e21aa05c95d4ef41c9f7117fe33"} Jul 28 00:46:48.832873 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.832844 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4dqfr" event={"ID":"7c8da112-99aa-4570-9c43-197db572fa95","Type":"ContainerStarted","Data":"7fd66da9b9861bf7cea0c8a4109a7b78bc94a36037a54513124aca8b8dfc8654"} Jul 28 00:46:48.851963 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.851907 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" event={"ID":"4f1301d4be0daa7e23534036d6f77d33","Type":"ContainerStarted","Data":"f84ce700d19d11c8e48105d8a24e0aa8c2d398dae0945aa52bbc7b3e05398c7d"} Jul 28 00:46:48.860946 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.860921 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6hd56" event={"ID":"7858a6fd-db8f-4492-b847-d19cec86528d","Type":"ContainerStarted","Data":"dfb79d67bc92523814ca7ef6d3dc0ee3b98f93c224588e75e708f1d6a8531ac3"} Jul 28 00:46:48.874210 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.874184 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-45kvm" event={"ID":"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06","Type":"ContainerStarted","Data":"ead28cc3f38b83d8ec348e6b2a9f751efbe704ce35d241d14da688d804580c3c"} Jul 28 00:46:48.886474 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.886360 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" event={"ID":"9c3f258e-cca5-450c-960a-53d3aad3953d","Type":"ContainerStarted","Data":"5db3e18ee832baa16a5aa9172f9419dcf719c89ce31444e4d707f9cd01850fa3"} Jul 28 00:46:48.905273 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.905147 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" event={"ID":"02567703-a907-465b-a016-386dd7208f74","Type":"ContainerStarted","Data":"4ddaad87a0ee45e0f7de3ae80d5d4cc495fc04883dc33d4c554685597caa013f"} Jul 28 00:46:48.923088 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.922822 2550 generic.go:356] "Generic (PLEG): container finished" podID="33cddadf0b9611b00f916f9e5eb1693c" containerID="69bab67af74717e73784a8aa64cd166d9044497404e8cdeb13a321d168faa4ae" exitCode=0 Jul 28 00:46:48.923088 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.922895 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" event={"ID":"33cddadf0b9611b00f916f9e5eb1693c","Type":"ContainerDied","Data":"69bab67af74717e73784a8aa64cd166d9044497404e8cdeb13a321d168faa4ae"} Jul 28 00:46:48.939175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.938046 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-173.ec2.internal" podStartSLOduration=2.938028734 podStartE2EDuration="2.938028734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:48.865924825 +0000 UTC m=+4.626772842" watchObservedRunningTime="2026-07-28 00:46:48.938028734 +0000 UTC m=+4.698876752" Jul 28 00:46:48.944496 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.944460 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wwgrl" event={"ID":"a38938d2-ef9d-43ec-91d9-575165164123","Type":"ContainerStarted","Data":"f5d8c1f7e11b3c8d4ed3f9199e1f8a74fba6d1465b86982b8c97b8c82877bfcf"} Jul 28 00:46:48.948992 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:48.948895 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" event={"ID":"eb91259a-5111-49b5-bf8d-f3d2404180c2","Type":"ContainerStarted","Data":"553e493fa0b81609ad1c7a7fcdabff59b43ecc690f5a95a4e5418b96af9653f0"} Jul 28 00:46:49.778752 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:49.778703 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:49.779213 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:49.778837 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:46:49.997785 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:49.997746 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" event={"ID":"33cddadf0b9611b00f916f9e5eb1693c","Type":"ContainerStarted","Data":"47857c75779f2e4c9c153dd616931fabb5aa2a3a11ed30b1eba867228892461d"} Jul 28 00:46:50.333738 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:50.333704 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:50.333925 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:50.333762 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:50.333925 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.333871 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:50.334056 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.333941 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs podName:07a0f217-710e-482b-a1bb-a358656d435f nodeName:}" failed. No retries permitted until 2026-07-28 00:46:54.333921895 +0000 UTC m=+10.094769889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs") pod "network-metrics-daemon-gddjn" (UID: "07a0f217-710e-482b-a1bb-a358656d435f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:50.334056 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.333973 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:50.334056 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.334038 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret podName:a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:54.334020331 +0000 UTC m=+10.094868327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret") pod "global-pull-secret-syncer-79x2c" (UID: "a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:50.434555 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:50.434479 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:50.434708 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.434684 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:46:50.434708 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.434702 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:46:50.434814 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.434714 2550 projected.go:196] Error preparing data for projected volume kube-api-access-nbjvp for pod openshift-network-diagnostics/network-check-target-8jqzf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:50.434814 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.434769 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp podName:57227b22-d127-4005-90b6-cc2d78d2fd38 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:54.434749895 +0000 UTC m=+10.195597899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjvp" (UniqueName: "kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp") pod "network-check-target-8jqzf" (UID: "57227b22-d127-4005-90b6-cc2d78d2fd38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:50.779209 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:50.779113 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:50.779652 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.779237 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:46:50.779878 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:50.779705 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:50.779878 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:50.779812 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:46:51.778368 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:51.778285 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:51.778556 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:51.778415 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:46:52.778174 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:52.778103 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:52.778597 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:52.778237 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:46:52.778597 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:52.778586 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:52.778722 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:52.778682 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:46:53.778531 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:53.778497 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:53.778964 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:53.778645 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:46:54.365482 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:54.365448 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:54.365678 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:54.365503 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:54.365678 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.365645 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:54.365789 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.365704 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret podName:a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:02.36568546 +0000 UTC m=+18.126533475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret") pod "global-pull-secret-syncer-79x2c" (UID: "a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:46:54.366098 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.366079 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:54.366216 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.366140 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs podName:07a0f217-710e-482b-a1bb-a358656d435f nodeName:}" failed. No retries permitted until 2026-07-28 00:47:02.366122823 +0000 UTC m=+18.126970818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs") pod "network-metrics-daemon-gddjn" (UID: "07a0f217-710e-482b-a1bb-a358656d435f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:46:54.465862 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:54.465795 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:54.466027 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.465996 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:46:54.466099 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.466029 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:46:54.466099 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.466042 2550 projected.go:196] Error preparing data for projected volume kube-api-access-nbjvp for pod openshift-network-diagnostics/network-check-target-8jqzf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:54.466099 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.466096 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp podName:57227b22-d127-4005-90b6-cc2d78d2fd38 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:02.466079607 +0000 UTC m=+18.226927619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjvp" (UniqueName: "kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp") pod "network-check-target-8jqzf" (UID: "57227b22-d127-4005-90b6-cc2d78d2fd38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:46:54.779922 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:54.779848 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:54.780383 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.779947 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:46:54.780383 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:54.780176 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:54.780383 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:54.780296 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:46:55.778774 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:55.778743 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:55.778914 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:55.778887 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:46:56.778793 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:56.778761 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:56.779275 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:56.778900 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:46:56.779275 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:56.778952 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:56.779275 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:56.779073 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:46:57.777814 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:57.777784 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:57.777966 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:57.777902 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:46:58.778209 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:58.778138 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:46:58.778632 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:58.778246 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:46:58.778632 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:58.778326 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:46:58.778632 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:58.778438 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:46:59.778474 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:46:59.778437 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:46:59.778880 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:46:59.778545 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:00.778225 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:00.778196 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:00.778417 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:00.778207 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:00.778417 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:00.778329 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:00.778522 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:00.778434 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:01.777887 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:01.777857 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:01.778139 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:01.777982 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:02.430305 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:02.430275 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:02.430305 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:02.430311 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:02.430724 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.430409 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:47:02.430724 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.430466 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs podName:07a0f217-710e-482b-a1bb-a358656d435f nodeName:}" failed. No retries permitted until 2026-07-28 00:47:18.430451552 +0000 UTC m=+34.191299547 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs") pod "network-metrics-daemon-gddjn" (UID: "07a0f217-710e-482b-a1bb-a358656d435f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:47:02.430724 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.430409 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:47:02.430724 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.430544 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret podName:a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:18.430532706 +0000 UTC m=+34.191380697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret") pod "global-pull-secret-syncer-79x2c" (UID: "a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:47:02.530817 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:02.530791 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:02.530959 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.530924 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:47:02.530959 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.530944 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:47:02.530959 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.530957 2550 projected.go:196] Error preparing data for projected volume kube-api-access-nbjvp for pod openshift-network-diagnostics/network-check-target-8jqzf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:47:02.531091 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.531017 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp podName:57227b22-d127-4005-90b6-cc2d78d2fd38 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:18.530991803 +0000 UTC m=+34.291839795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjvp" (UniqueName: "kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp") pod "network-check-target-8jqzf" (UID: "57227b22-d127-4005-90b6-cc2d78d2fd38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:47:02.778737 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:02.778658 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:02.778879 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.778787 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:02.778879 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:02.778827 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:02.778969 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:02.778907 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:03.778751 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:03.778720 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:03.779155 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:03.778877 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:04.779672 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:04.779639 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:04.780118 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:04.779746 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:04.780118 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:04.779804 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:04.780118 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:04.779862 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:05.778042 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:05.777997 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:05.778215 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:05.778142 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:06.778076 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:06.777781 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:06.778741 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:06.777795 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:06.778741 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:06.778143 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:06.778741 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:06.778223 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:07.030654 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.030580 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" event={"ID":"02567703-a907-465b-a016-386dd7208f74","Type":"ContainerStarted","Data":"f4975ab0373ccd486c2a2fefc4f3a83d34503540502d368dd765be9afd977cbb"} Jul 28 00:47:07.031958 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.031921 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wwgrl" event={"ID":"a38938d2-ef9d-43ec-91d9-575165164123","Type":"ContainerStarted","Data":"fd08044254f57e7cd390d88ace8c5d5880980e4d103b4c9cf8386daa9be62787"} Jul 28 00:47:07.034811 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.034789 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" event={"ID":"eb91259a-5111-49b5-bf8d-f3d2404180c2","Type":"ContainerStarted","Data":"ce75dc9802c773ce22db2fba820390f1318d8e209a0dc29964fe3376c5e0d651"} Jul 28 00:47:07.034892 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.034862 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" event={"ID":"eb91259a-5111-49b5-bf8d-f3d2404180c2","Type":"ContainerStarted","Data":"53d7540ca28994bff7fbd1d072df6941606580e702db05b6e135986e34a38806"} Jul 28 00:47:07.034938 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.034918 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" event={"ID":"eb91259a-5111-49b5-bf8d-f3d2404180c2","Type":"ContainerStarted","Data":"be4b55b1b967d984c4b7d7c13529b094700b4faa56771f53c2144a4750d88f25"} Jul 28 00:47:07.034981 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.034966 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" event={"ID":"eb91259a-5111-49b5-bf8d-f3d2404180c2","Type":"ContainerStarted","Data":"214c5082687fb0c22b343a0e363dcee39d24402a36eb1ca08d5852bcc0444bd3"} Jul 28 00:47:07.035040 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.035025 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" event={"ID":"eb91259a-5111-49b5-bf8d-f3d2404180c2","Type":"ContainerStarted","Data":"fa5c61174d44dec39789e8f952b9fde8fe3b648edae4aa76a9717f1876674567"} Jul 28 00:47:07.035087 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.035055 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" event={"ID":"eb91259a-5111-49b5-bf8d-f3d2404180c2","Type":"ContainerStarted","Data":"34cfaf0954324a64f724f28adb8cf094f06d00b3274c5b7697a5d833f8ed74b4"} Jul 28 00:47:07.036226 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.036194 2550 generic.go:356] "Generic (PLEG): container finished" podID="a14ecd2d-fc55-40dd-85fa-6152eb2ce28c" containerID="bddb9fef83f4010720d509fb5be029c4109836caef3271e3cd136808ce6001bd" exitCode=0 Jul 28 00:47:07.036311 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.036253 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6sh29" event={"ID":"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c","Type":"ContainerDied","Data":"bddb9fef83f4010720d509fb5be029c4109836caef3271e3cd136808ce6001bd"} Jul 28 00:47:07.037657 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.037636 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-svcsw" event={"ID":"4dfca9b2-d814-4d0f-8754-eec2a38a19fe","Type":"ContainerStarted","Data":"8570ce54c54c42e2573498c7f0d098db7ad51cde17d24cfffe03219f18565acb"} Jul 28 00:47:07.039058 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.039030 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4dqfr" event={"ID":"7c8da112-99aa-4570-9c43-197db572fa95","Type":"ContainerStarted","Data":"56ebba948b729a7cf1c8066af582029be356996dd1aa668e833523d9cb48fd21"} Jul 28 00:47:07.040345 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.040321 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6hd56" event={"ID":"7858a6fd-db8f-4492-b847-d19cec86528d","Type":"ContainerStarted","Data":"da202ccadc75743de1ba481fd03654a9ac47aeba6ee5cacf3b985ed48fe879e2"} Jul 28 00:47:07.041566 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.041549 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" event={"ID":"9c3f258e-cca5-450c-960a-53d3aad3953d","Type":"ContainerStarted","Data":"a92f655f5575096b4a64df09241ba209cd60576852c3a9824b737ce70dea15de"} Jul 28 00:47:07.049939 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.049904 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bb8x9" podStartSLOduration=3.701573868 podStartE2EDuration="22.04989288s" totalImagesPullingTime="18.348319012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:07.049681991 +0000 UTC m=+22.810530008" watchObservedRunningTime="2026-07-28 00:47:07.04989288 +0000 UTC m=+22.810740919" Jul 28 00:47:07.050045 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.049988 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-173.ec2.internal" podStartSLOduration=21.049983774 podStartE2EDuration="21.049983774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:50.01216688 +0000 UTC m=+5.773015070" watchObservedRunningTime="2026-07-28 00:47:07.049983774 +0000 UTC m=+22.810831787" Jul 28 00:47:07.063316 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.063285 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wwgrl" podStartSLOduration=3.723873847 podStartE2EDuration="22.063265192s" totalImagesPullingTime="18.339391345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:07.06293915 +0000 UTC m=+22.823787162" watchObservedRunningTime="2026-07-28 00:47:07.063265192 +0000 UTC m=+22.824113204" Jul 28 00:47:07.077174 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.077134 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4dqfr" podStartSLOduration=3.728629598 podStartE2EDuration="22.077114728s" totalImagesPullingTime="18.34848513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:07.076477157 +0000 UTC m=+22.837325171" watchObservedRunningTime="2026-07-28 00:47:07.077114728 +0000 UTC m=+22.837962740" Jul 28 00:47:07.089987 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.089954 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-svcsw" podStartSLOduration=11.58627482 podStartE2EDuration="22.089942293s" totalImagesPullingTime="10.503667473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:07.08944271 +0000 UTC m=+22.850290723" watchObservedRunningTime="2026-07-28 00:47:07.089942293 +0000 UTC m=+22.850790307" Jul 28 00:47:07.673224 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.673196 2550 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 00:47:07.778333 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:07.778275 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:07.778748 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:07.778400 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:08.045135 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:08.045055 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-45kvm" event={"ID":"9153a9e6-ab69-4eeb-b4dd-dbd5a6ab1a06","Type":"ContainerStarted","Data":"184093e41d6b9ebfc41a92d545d8723d93e25309394799c1d2bec8990cd0673c"} Jul 28 00:47:08.047141 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:08.047113 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" event={"ID":"9c3f258e-cca5-450c-960a-53d3aad3953d","Type":"ContainerStarted","Data":"f3f3567ba510e1c4826f41dbc9cfe1e41b4c2819b2fc0758af25555e5bace865"} Jul 28 00:47:08.060466 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:08.060421 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-45kvm" podStartSLOduration=4.723035705 podStartE2EDuration="23.060403306s" totalImagesPullingTime="18.337367601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:08.060294173 +0000 UTC m=+23.821142186" watchObservedRunningTime="2026-07-28 00:47:08.060403306 +0000 UTC m=+23.821251321" Jul 28 00:47:08.061578 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:08.061533 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6hd56" podStartSLOduration=4.672481149 podStartE2EDuration="23.061522825s" totalImagesPullingTime="18.389041676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:07.125674119 +0000 UTC m=+22.886522133" watchObservedRunningTime="2026-07-28 00:47:08.061522825 +0000 UTC m=+23.822370839" Jul 28 00:47:08.669171 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:08.669072 2550 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T00:47:07.673219975Z","UUID":"fb337a25-0b00-4606-a9bb-883a5632ceee","Handler":null,"Name":"","Endpoint":""} Jul 28 00:47:08.670951 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:08.670928 2550 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 00:47:08.671191 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:08.670972 2550 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 00:47:08.777975 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:08.777946 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:08.778123 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:08.777947 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:08.778123 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:08.778074 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:08.778240 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:08.778149 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:09.052075 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:09.051786 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" event={"ID":"9c3f258e-cca5-450c-960a-53d3aad3953d","Type":"ContainerStarted","Data":"e6346cf902f59964b0531e47b0236214f8788e217904435887aa47407d6d9739"} Jul 28 00:47:09.057565 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:09.057537 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" event={"ID":"eb91259a-5111-49b5-bf8d-f3d2404180c2","Type":"ContainerStarted","Data":"3d86e5ea4ba3074bb0faf7826086dec0c72ecb620d7129a217521e90666bb228"} Jul 28 00:47:09.069562 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:09.069518 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vfrg2" podStartSLOduration=3.449189173 podStartE2EDuration="24.069505438s" totalImagesPullingTime="20.620316265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:09.069464399 +0000 UTC m=+24.830312419" watchObservedRunningTime="2026-07-28 00:47:09.069505438 +0000 UTC m=+24.830353451" Jul 28 00:47:09.778312 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:09.778284 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:09.778509 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:09.778407 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:10.778119 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:10.778085 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:10.778698 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:10.778209 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:10.778698 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:10.778281 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:10.778698 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:10.778384 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:11.408521 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:11.408319 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:47:11.408787 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:11.408768 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:47:11.778184 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:11.778132 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:11.778669 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:11.778225 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:12.068899 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.068827 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" event={"ID":"eb91259a-5111-49b5-bf8d-f3d2404180c2","Type":"ContainerStarted","Data":"3f1e435771fa81fb7bb9e2430cf0d0c0efeb8092441e2fc5bd5dbd598f100be6"} Jul 28 00:47:12.069100 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.069081 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:47:12.069199 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.069158 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:47:12.070705 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.070680 2550 generic.go:356] "Generic (PLEG): container finished" podID="a14ecd2d-fc55-40dd-85fa-6152eb2ce28c" containerID="98b1affeee315b8b17f8bd7c66098320df811e2fa50114b778016b5d0a09764c" exitCode=0 Jul 28 00:47:12.070812 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.070763 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6sh29" event={"ID":"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c","Type":"ContainerDied","Data":"98b1affeee315b8b17f8bd7c66098320df811e2fa50114b778016b5d0a09764c"} Jul 28 00:47:12.071048 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.071035 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:47:12.072033 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.071411 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-svcsw" Jul 28 00:47:12.085551 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.085531 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:47:12.097415 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.097378 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" podStartSLOduration=8.68794121 podStartE2EDuration="27.097365032s" totalImagesPullingTime="18.409423822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:12.095860576 +0000 UTC m=+27.856708590" watchObservedRunningTime="2026-07-28 00:47:12.097365032 +0000 UTC m=+27.858213045" Jul 28 00:47:12.778926 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.778892 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:12.779362 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:12.778903 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:12.779362 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:12.779071 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:12.779362 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:12.779130 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:13.072880 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:13.072859 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:47:13.087243 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:13.087220 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:47:13.778653 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:13.778627 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:13.778810 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:13.778732 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:14.076596 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:14.076532 2550 generic.go:356] "Generic (PLEG): container finished" podID="a14ecd2d-fc55-40dd-85fa-6152eb2ce28c" containerID="825ca0257e461b0b9458b56b634e403b53e88ccc393dc275ac0f77fb76525209" exitCode=0 Jul 28 00:47:14.076939 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:14.076614 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6sh29" event={"ID":"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c","Type":"ContainerDied","Data":"825ca0257e461b0b9458b56b634e403b53e88ccc393dc275ac0f77fb76525209"} Jul 28 00:47:14.779284 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:14.779260 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:14.779428 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:14.779342 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:14.779474 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:14.779430 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:14.779532 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:14.779514 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:15.081049 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:15.081024 2550 generic.go:356] "Generic (PLEG): container finished" podID="a14ecd2d-fc55-40dd-85fa-6152eb2ce28c" containerID="cd0f1ef877e271f097fb01542b1c41216fabf209792bdc2b75ccf4e7faaea25c" exitCode=0 Jul 28 00:47:15.081387 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:15.081105 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6sh29" event={"ID":"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c","Type":"ContainerDied","Data":"cd0f1ef877e271f097fb01542b1c41216fabf209792bdc2b75ccf4e7faaea25c"} Jul 28 00:47:15.777858 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:15.777829 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:15.778038 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:15.777938 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:16.778179 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:16.777970 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:16.778619 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:16.778065 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:16.778619 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:16.778270 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:16.778619 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:16.778355 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:17.778540 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:17.778507 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:17.778997 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:17.778640 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:18.388554 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:18.388041 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jqzf"] Jul 28 00:47:18.388554 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:18.388150 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:18.388554 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.388254 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:18.388821 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:18.388776 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-79x2c"] Jul 28 00:47:18.389055 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:18.388919 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:18.389055 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.389039 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:18.389671 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:18.389652 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gddjn"] Jul 28 00:47:18.389759 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:18.389744 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:18.389856 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.389840 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:18.457978 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:18.457904 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:18.457978 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:18.457949 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:18.458169 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.458080 2550 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:47:18.458169 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.458142 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs podName:07a0f217-710e-482b-a1bb-a358656d435f nodeName:}" failed. No retries permitted until 2026-07-28 00:47:50.458128106 +0000 UTC m=+66.218976101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs") pod "network-metrics-daemon-gddjn" (UID: "07a0f217-710e-482b-a1bb-a358656d435f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 00:47:18.458256 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.458080 2550 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 00:47:18.458256 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.458230 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret podName:a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:50.45821325 +0000 UTC m=+66.219061253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret") pod "global-pull-secret-syncer-79x2c" (UID: "a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7") : object "kube-system"/"original-pull-secret" not registered Jul 28 00:47:18.559716 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:18.559352 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:18.559716 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.559552 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 00:47:18.559716 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.559572 2550 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 00:47:18.559716 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.559585 2550 projected.go:196] Error preparing data for projected volume kube-api-access-nbjvp for pod openshift-network-diagnostics/network-check-target-8jqzf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:47:18.559716 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:18.559646 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp podName:57227b22-d127-4005-90b6-cc2d78d2fd38 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:50.559626172 +0000 UTC m=+66.320474176 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbjvp" (UniqueName: "kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp") pod "network-check-target-8jqzf" (UID: "57227b22-d127-4005-90b6-cc2d78d2fd38") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 00:47:19.778162 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:19.778118 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:19.778864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:19.778119 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:19.778864 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:19.778236 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:19.778864 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:19.778355 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:20.095486 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:20.095456 2550 generic.go:356] "Generic (PLEG): container finished" podID="a14ecd2d-fc55-40dd-85fa-6152eb2ce28c" containerID="caf30ed28b74cb8a1706725988052f7934931a46459b78b7e34ddbcda9f7ae53" exitCode=0 Jul 28 00:47:20.095584 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:20.095499 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6sh29" event={"ID":"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c","Type":"ContainerDied","Data":"caf30ed28b74cb8a1706725988052f7934931a46459b78b7e34ddbcda9f7ae53"} Jul 28 00:47:20.778124 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:20.778063 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:20.778236 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:20.778150 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79x2c" podUID="a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7" Jul 28 00:47:21.100318 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:21.100287 2550 generic.go:356] "Generic (PLEG): container finished" podID="a14ecd2d-fc55-40dd-85fa-6152eb2ce28c" containerID="8eba12f41222b1b3cc3142fb9d21926a5da27d29ee52dcef76c49221f869730f" exitCode=0 Jul 28 00:47:21.100463 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:21.100324 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6sh29" event={"ID":"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c","Type":"ContainerDied","Data":"8eba12f41222b1b3cc3142fb9d21926a5da27d29ee52dcef76c49221f869730f"} Jul 28 00:47:21.778701 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:21.778533 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:21.779080 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:21.778533 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:21.779080 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:21.778771 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-gddjn" podUID="07a0f217-710e-482b-a1bb-a358656d435f" Jul 28 00:47:21.779080 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:21.778880 2550 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jqzf" podUID="57227b22-d127-4005-90b6-cc2d78d2fd38" Jul 28 00:47:22.083659 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.083633 2550 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-173.ec2.internal" event="NodeReady" Jul 28 00:47:22.083836 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.083761 2550 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 00:47:22.105150 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.105124 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6sh29" event={"ID":"a14ecd2d-fc55-40dd-85fa-6152eb2ce28c","Type":"ContainerStarted","Data":"4288275498e24970e2a5bed549818232367056529f685853c900f45eef4c089b"} Jul 28 00:47:22.118432 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.118406 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-559f46ddb8-kkksq"] Jul 28 00:47:22.120235 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.120218 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.122531 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.122510 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qz9w8\"" Jul 28 00:47:22.122641 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.122511 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 00:47:22.122641 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.122583 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 00:47:22.122881 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.122863 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 00:47:22.129612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.129592 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 00:47:22.130109 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.130074 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6sh29" podStartSLOduration=8.597769184 podStartE2EDuration="37.130060355s" totalImagesPullingTime="28.532291171s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:22.128934078 +0000 UTC m=+37.889790055" watchObservedRunningTime="2026-07-28 00:47:22.130060355 +0000 UTC m=+37.890908363" Jul 28 00:47:22.135596 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.135578 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qsxxt"] Jul 28 00:47:22.137240 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.137220 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.141174 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.141108 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 00:47:22.141373 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.141353 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-54rrh\"" Jul 28 00:47:22.141824 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.141671 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 00:47:22.141824 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.141671 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-559f46ddb8-kkksq"] Jul 28 00:47:22.142487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.142469 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8hn84"] Jul 28 00:47:22.144242 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.144224 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8hn84" Jul 28 00:47:22.146518 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.146495 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nzvbc\"" Jul 28 00:47:22.146663 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.146649 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 00:47:22.146951 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.146928 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 00:47:22.147071 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.146931 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 00:47:22.148957 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.148935 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qsxxt"] Jul 28 00:47:22.155980 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.155951 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8hn84"] Jul 28 00:47:22.165998 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.165979 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jzhk9"] Jul 28 00:47:22.167827 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.167812 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.170139 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.170124 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f27qr\"" Jul 28 00:47:22.171682 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.171657 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 00:47:22.171794 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.171774 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 00:47:22.172344 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.172320 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 00:47:22.172548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.172529 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 00:47:22.180222 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.180200 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jzhk9"] Jul 28 00:47:22.290536 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290513 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2lm7\" (UniqueName: \"kubernetes.io/projected/a98530fe-6e13-4892-af34-8d96fb00e3f7-kube-api-access-v2lm7\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.290649 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290555 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndk5v\" (UniqueName: \"kubernetes.io/projected/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-kube-api-access-ndk5v\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.290649 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290576 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82f9b919-9e2e-4a87-9049-4691045afa53-ca-trust-extracted\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.290649 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290617 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4-cert\") pod \"ingress-canary-8hn84\" (UID: \"f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4\") " pod="openshift-ingress-canary/ingress-canary-8hn84" Jul 28 00:47:22.290786 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290735 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-crio-socket\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.290786 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290753 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a98530fe-6e13-4892-af34-8d96fb00e3f7-tmp-dir\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.290786 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290770 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.290932 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290792 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmnsw\" (UniqueName: \"kubernetes.io/projected/f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4-kube-api-access-cmnsw\") pod \"ingress-canary-8hn84\" (UID: \"f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4\") " pod="openshift-ingress-canary/ingress-canary-8hn84" Jul 28 00:47:22.290932 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290847 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-registry-certificates\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.290932 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290885 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-image-registry-private-configuration\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.290932 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290923 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.291149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290953 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-bound-sa-token\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.291149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290976 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a98530fe-6e13-4892-af34-8d96fb00e3f7-config-volume\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.291149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.290992 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a98530fe-6e13-4892-af34-8d96fb00e3f7-metrics-tls\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.291149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.291048 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-data-volume\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.291149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.291081 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-registry-tls\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.291149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.291144 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-installation-pull-secrets\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.291341 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.291167 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpstk\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-kube-api-access-zpstk\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.291341 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.291193 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-trusted-ca\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.391875 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.391822 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndk5v\" (UniqueName: \"kubernetes.io/projected/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-kube-api-access-ndk5v\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.391875 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.391852 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82f9b919-9e2e-4a87-9049-4691045afa53-ca-trust-extracted\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.392103 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.391877 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4-cert\") pod \"ingress-canary-8hn84\" (UID: \"f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4\") " pod="openshift-ingress-canary/ingress-canary-8hn84" Jul 28 00:47:22.392103 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392034 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-crio-socket\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.392103 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392054 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a98530fe-6e13-4892-af34-8d96fb00e3f7-tmp-dir\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.392103 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392079 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.392103 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392097 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmnsw\" (UniqueName: \"kubernetes.io/projected/f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4-kube-api-access-cmnsw\") pod \"ingress-canary-8hn84\" (UID: \"f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4\") " pod="openshift-ingress-canary/ingress-canary-8hn84" Jul 28 00:47:22.392320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392116 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-registry-certificates\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.392320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392146 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-image-registry-private-configuration\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.392320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392177 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.392320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392198 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-bound-sa-token\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.392320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392220 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a98530fe-6e13-4892-af34-8d96fb00e3f7-config-volume\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.392320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392241 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a98530fe-6e13-4892-af34-8d96fb00e3f7-metrics-tls\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.392320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392275 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82f9b919-9e2e-4a87-9049-4691045afa53-ca-trust-extracted\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.392320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392281 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-data-volume\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.392585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392320 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-registry-tls\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.392585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392360 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-installation-pull-secrets\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.392585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392389 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpstk\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-kube-api-access-zpstk\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.392585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392416 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-trusted-ca\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.392585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392442 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lm7\" (UniqueName: \"kubernetes.io/projected/a98530fe-6e13-4892-af34-8d96fb00e3f7-kube-api-access-v2lm7\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.392585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392555 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-data-volume\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.392585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392583 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-crio-socket\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.392866 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392416 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a98530fe-6e13-4892-af34-8d96fb00e3f7-tmp-dir\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.393017 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.392971 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-registry-certificates\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.393402 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.393289 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.393402 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.393390 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-trusted-ca\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.393535 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.393468 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a98530fe-6e13-4892-af34-8d96fb00e3f7-config-volume\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.396510 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.396476 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a98530fe-6e13-4892-af34-8d96fb00e3f7-metrics-tls\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.396607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.396478 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.396607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.396517 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-image-registry-private-configuration\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.396697 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.396679 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-registry-tls\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.396739 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.396698 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4-cert\") pod \"ingress-canary-8hn84\" (UID: \"f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4\") " pod="openshift-ingress-canary/ingress-canary-8hn84" Jul 28 00:47:22.396856 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.396836 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-installation-pull-secrets\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.399697 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.399673 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2lm7\" (UniqueName: \"kubernetes.io/projected/a98530fe-6e13-4892-af34-8d96fb00e3f7-kube-api-access-v2lm7\") pod \"dns-default-qsxxt\" (UID: \"a98530fe-6e13-4892-af34-8d96fb00e3f7\") " pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.400225 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.400202 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndk5v\" (UniqueName: \"kubernetes.io/projected/a8efbf8b-4dbb-45b5-9443-0553e28ffe10-kube-api-access-ndk5v\") pod \"insights-runtime-extractor-jzhk9\" (UID: \"a8efbf8b-4dbb-45b5-9443-0553e28ffe10\") " pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.400317 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.400301 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-bound-sa-token\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.400491 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.400465 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpstk\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-kube-api-access-zpstk\") pod \"image-registry-559f46ddb8-kkksq\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.400744 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.400730 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmnsw\" (UniqueName: \"kubernetes.io/projected/f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4-kube-api-access-cmnsw\") pod \"ingress-canary-8hn84\" (UID: \"f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4\") " pod="openshift-ingress-canary/ingress-canary-8hn84" Jul 28 00:47:22.431521 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.431506 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:22.448038 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.448017 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:22.456071 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.456055 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8hn84" Jul 28 00:47:22.478935 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.478915 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jzhk9" Jul 28 00:47:22.592502 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.592335 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-559f46ddb8-kkksq"] Jul 28 00:47:22.596654 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:22.596629 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f9b919_9e2e_4a87_9049_4691045afa53.slice/crio-6b079d9534a62dafdb5ebe389ba1a83021fae5dfcb4b2aa560be6ca7ba22f9f5 WatchSource:0}: Error finding container 6b079d9534a62dafdb5ebe389ba1a83021fae5dfcb4b2aa560be6ca7ba22f9f5: Status 404 returned error can't find the container with id 6b079d9534a62dafdb5ebe389ba1a83021fae5dfcb4b2aa560be6ca7ba22f9f5 Jul 28 00:47:22.606636 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.606612 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qsxxt"] Jul 28 00:47:22.611144 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:22.611114 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98530fe_6e13_4892_af34_8d96fb00e3f7.slice/crio-813185abaf5edb58dfdde3f4f353482ec758bcc1367b1e2d9f80e126cbe0cc3f WatchSource:0}: Error finding container 813185abaf5edb58dfdde3f4f353482ec758bcc1367b1e2d9f80e126cbe0cc3f: Status 404 returned error can't find the container with id 813185abaf5edb58dfdde3f4f353482ec758bcc1367b1e2d9f80e126cbe0cc3f Jul 28 00:47:22.616845 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.616801 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8hn84"] Jul 28 00:47:22.627046 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:22.627017 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9c96ebe_d995_41ae_a3cb_2ebd07bcdbf4.slice/crio-5b436572f2480dee7bf0cbd3d03e7c5ea317b88ef72164dd21b0cefa2b8d87a5 WatchSource:0}: Error finding container 5b436572f2480dee7bf0cbd3d03e7c5ea317b88ef72164dd21b0cefa2b8d87a5: Status 404 returned error can't find the container with id 5b436572f2480dee7bf0cbd3d03e7c5ea317b88ef72164dd21b0cefa2b8d87a5 Jul 28 00:47:22.629178 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.629157 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jzhk9"] Jul 28 00:47:22.632319 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:22.632298 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8efbf8b_4dbb_45b5_9443_0553e28ffe10.slice/crio-064e1ecfebf0bc296e568b644af8c4c5170d171f62c39b6424390cd19231c3b5 WatchSource:0}: Error finding container 064e1ecfebf0bc296e568b644af8c4c5170d171f62c39b6424390cd19231c3b5: Status 404 returned error can't find the container with id 064e1ecfebf0bc296e568b644af8c4c5170d171f62c39b6424390cd19231c3b5 Jul 28 00:47:22.778278 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.778256 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:22.780603 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:22.780581 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 00:47:23.110540 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.110433 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" event={"ID":"82f9b919-9e2e-4a87-9049-4691045afa53","Type":"ContainerStarted","Data":"f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0"} Jul 28 00:47:23.110701 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.110570 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" event={"ID":"82f9b919-9e2e-4a87-9049-4691045afa53","Type":"ContainerStarted","Data":"6b079d9534a62dafdb5ebe389ba1a83021fae5dfcb4b2aa560be6ca7ba22f9f5"} Jul 28 00:47:23.110786 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.110761 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:23.111909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.111885 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzhk9" event={"ID":"a8efbf8b-4dbb-45b5-9443-0553e28ffe10","Type":"ContainerStarted","Data":"064e1ecfebf0bc296e568b644af8c4c5170d171f62c39b6424390cd19231c3b5"} Jul 28 00:47:23.113670 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.113638 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8hn84" event={"ID":"f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4","Type":"ContainerStarted","Data":"5b436572f2480dee7bf0cbd3d03e7c5ea317b88ef72164dd21b0cefa2b8d87a5"} Jul 28 00:47:23.115334 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.115310 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsxxt" event={"ID":"a98530fe-6e13-4892-af34-8d96fb00e3f7","Type":"ContainerStarted","Data":"813185abaf5edb58dfdde3f4f353482ec758bcc1367b1e2d9f80e126cbe0cc3f"} Jul 28 00:47:23.135532 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.135487 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" podStartSLOduration=61.135473127 podStartE2EDuration="1m1.135473127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:23.130187514 +0000 UTC m=+38.891035528" watchObservedRunningTime="2026-07-28 00:47:23.135473127 +0000 UTC m=+38.896321139" Jul 28 00:47:23.779719 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.778450 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:23.779719 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.778885 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:23.782452 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.782427 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mkfth\"" Jul 28 00:47:23.783114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.782717 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 00:47:23.783114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.782900 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 00:47:23.783114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.782943 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kdcl9\"" Jul 28 00:47:23.783114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:23.783099 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 00:47:30.136166 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:30.136116 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzhk9" event={"ID":"a8efbf8b-4dbb-45b5-9443-0553e28ffe10","Type":"ContainerStarted","Data":"692e28f86da2cd87d0dcc52c5c95292a489eebe41fcf15054caf7edfa23e4551"} Jul 28 00:47:30.137713 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:30.137684 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8hn84" event={"ID":"f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4","Type":"ContainerStarted","Data":"0249b7f70816cfba651f51d59491896fe82f4bca2bcc0b988bc5ebd05d1d5449"} Jul 28 00:47:30.139561 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:30.139536 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsxxt" event={"ID":"a98530fe-6e13-4892-af34-8d96fb00e3f7","Type":"ContainerStarted","Data":"dc1d12de180d97cdef7b097d9cb20c7c3c4ce7bf1d237518fcb6d37b684cc601"} Jul 28 00:47:30.139668 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:30.139583 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qsxxt" event={"ID":"a98530fe-6e13-4892-af34-8d96fb00e3f7","Type":"ContainerStarted","Data":"c1042f9517f2d6ef23af7b485a648025291b42aed78bff2fe362a30d58208fc0"} Jul 28 00:47:30.139724 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:30.139682 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:30.152800 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:30.152754 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8hn84" podStartSLOduration=1.743657461 podStartE2EDuration="8.152736448s" totalImagesPullingTime="6.409078987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:30.151628907 +0000 UTC m=+45.912476920" watchObservedRunningTime="2026-07-28 00:47:30.152736448 +0000 UTC m=+45.913584462" Jul 28 00:47:30.168179 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:30.168127 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qsxxt" podStartSLOduration=1.743953415 podStartE2EDuration="8.16811243s" totalImagesPullingTime="6.424159015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:30.167237549 +0000 UTC m=+45.928085573" watchObservedRunningTime="2026-07-28 00:47:30.16811243 +0000 UTC m=+45.928960445" Jul 28 00:47:32.631992 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.631959 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-478rq"] Jul 28 00:47:32.672737 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.672702 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-478rq"] Jul 28 00:47:32.672882 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.672827 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.675443 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.675414 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 00:47:32.675659 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.675643 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 00:47:32.675747 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.675692 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bt2gw\"" Jul 28 00:47:32.676491 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.676472 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 00:47:32.676784 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.676671 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 00:47:32.676784 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.676706 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 00:47:32.770941 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.770906 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad4642d9-bfe2-443e-b139-444ae3f216ce-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.771110 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.770992 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad4642d9-bfe2-443e-b139-444ae3f216ce-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.771110 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.771050 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxg6g\" (UniqueName: \"kubernetes.io/projected/ad4642d9-bfe2-443e-b139-444ae3f216ce-kube-api-access-xxg6g\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.771110 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.771075 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad4642d9-bfe2-443e-b139-444ae3f216ce-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.872396 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.872355 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad4642d9-bfe2-443e-b139-444ae3f216ce-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.872547 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.872442 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad4642d9-bfe2-443e-b139-444ae3f216ce-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.872547 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.872477 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxg6g\" (UniqueName: \"kubernetes.io/projected/ad4642d9-bfe2-443e-b139-444ae3f216ce-kube-api-access-xxg6g\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.872547 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.872510 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad4642d9-bfe2-443e-b139-444ae3f216ce-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.873122 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:32.873091 2550 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 28 00:47:32.873223 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:32.873169 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad4642d9-bfe2-443e-b139-444ae3f216ce-prometheus-operator-tls podName:ad4642d9-bfe2-443e-b139-444ae3f216ce nodeName:}" failed. No retries permitted until 2026-07-28 00:47:33.373149334 +0000 UTC m=+49.133997339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ad4642d9-bfe2-443e-b139-444ae3f216ce-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-478rq" (UID: "ad4642d9-bfe2-443e-b139-444ae3f216ce") : secret "prometheus-operator-tls" not found Jul 28 00:47:32.873303 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.873281 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ad4642d9-bfe2-443e-b139-444ae3f216ce-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.877000 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.876977 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ad4642d9-bfe2-443e-b139-444ae3f216ce-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:32.884102 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:32.883810 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxg6g\" (UniqueName: \"kubernetes.io/projected/ad4642d9-bfe2-443e-b139-444ae3f216ce-kube-api-access-xxg6g\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:33.375660 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:33.375629 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad4642d9-bfe2-443e-b139-444ae3f216ce-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:33.377797 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:33.377773 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ad4642d9-bfe2-443e-b139-444ae3f216ce-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-478rq\" (UID: \"ad4642d9-bfe2-443e-b139-444ae3f216ce\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:33.583166 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:33.583143 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" Jul 28 00:47:33.786822 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:33.786795 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-478rq"] Jul 28 00:47:33.789434 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:33.789405 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4642d9_bfe2_443e_b139_444ae3f216ce.slice/crio-e1b9526ab44b1e88254693a9ede22535897356221c6a2aa747d2718ba8cab157 WatchSource:0}: Error finding container e1b9526ab44b1e88254693a9ede22535897356221c6a2aa747d2718ba8cab157: Status 404 returned error can't find the container with id e1b9526ab44b1e88254693a9ede22535897356221c6a2aa747d2718ba8cab157 Jul 28 00:47:34.153211 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:34.153177 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" event={"ID":"ad4642d9-bfe2-443e-b139-444ae3f216ce","Type":"ContainerStarted","Data":"e1b9526ab44b1e88254693a9ede22535897356221c6a2aa747d2718ba8cab157"} Jul 28 00:47:34.154801 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:34.154780 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzhk9" event={"ID":"a8efbf8b-4dbb-45b5-9443-0553e28ffe10","Type":"ContainerStarted","Data":"68fc3688a3f56b597d06a77064fba845f6c285ff348bc0826198a6e05f19c334"} Jul 28 00:47:36.164795 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:36.164757 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" event={"ID":"ad4642d9-bfe2-443e-b139-444ae3f216ce","Type":"ContainerStarted","Data":"663f7e12fd5f119a62dd2edafddc851d3819ad14433bc582c92c6bfbdd5e67f5"} Jul 28 00:47:36.165362 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:36.164823 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" event={"ID":"ad4642d9-bfe2-443e-b139-444ae3f216ce","Type":"ContainerStarted","Data":"3eafa7da6e7e7443757eae90c966ecf3570ca094be308d3e277cf08880e22646"} Jul 28 00:47:36.181912 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:36.181866 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-478rq" podStartSLOduration=2.654932627 podStartE2EDuration="4.181846806s" totalImagesPullingTime="1.526914179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:36.181717399 +0000 UTC m=+51.942565426" watchObservedRunningTime="2026-07-28 00:47:36.181846806 +0000 UTC m=+51.942694821" Jul 28 00:47:38.022083 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.022053 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wn7mt"] Jul 28 00:47:38.046374 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.046348 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.048940 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.048914 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 00:47:38.049057 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.048944 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 00:47:38.049057 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.048960 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hpvwn\"" Jul 28 00:47:38.049169 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.049054 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 00:47:38.111356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.111303 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fshbl\" (UniqueName: \"kubernetes.io/projected/a106d45c-6d50-4df5-ae35-fce07bd9bf73-kube-api-access-fshbl\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.111356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.111330 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a106d45c-6d50-4df5-ae35-fce07bd9bf73-metrics-client-ca\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.111356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.111348 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a106d45c-6d50-4df5-ae35-fce07bd9bf73-sys\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.111512 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.111370 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-tls\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.111512 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.111420 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.111512 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.111450 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a106d45c-6d50-4df5-ae35-fce07bd9bf73-root\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.111512 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.111484 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-textfile\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.111512 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.111502 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-accelerators-collector-config\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.111707 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.111518 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-wtmp\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.172294 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.172269 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jzhk9" event={"ID":"a8efbf8b-4dbb-45b5-9443-0553e28ffe10","Type":"ContainerStarted","Data":"4c6b67ec91394ff71c0f0202eb019d89c9009f7dcd37e4bafc9bb2a467c5f9fa"} Jul 28 00:47:38.188871 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.188834 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jzhk9" podStartSLOduration=1.231242677 podStartE2EDuration="16.188819671s" totalImagesPullingTime="14.957576994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:47:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:38.188418754 +0000 UTC m=+53.949266768" watchObservedRunningTime="2026-07-28 00:47:38.188819671 +0000 UTC m=+53.949667684" Jul 28 00:47:38.212087 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212065 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fshbl\" (UniqueName: \"kubernetes.io/projected/a106d45c-6d50-4df5-ae35-fce07bd9bf73-kube-api-access-fshbl\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212099 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a106d45c-6d50-4df5-ae35-fce07bd9bf73-metrics-client-ca\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212121 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a106d45c-6d50-4df5-ae35-fce07bd9bf73-sys\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212146 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-tls\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212382 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212179 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212382 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212203 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a106d45c-6d50-4df5-ae35-fce07bd9bf73-root\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212382 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212263 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-textfile\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212382 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212279 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a106d45c-6d50-4df5-ae35-fce07bd9bf73-root\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212382 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212286 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-accelerators-collector-config\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212382 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212334 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-wtmp\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212674 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212200 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a106d45c-6d50-4df5-ae35-fce07bd9bf73-sys\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212674 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212464 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-wtmp\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212764 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212695 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-textfile\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212764 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212704 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a106d45c-6d50-4df5-ae35-fce07bd9bf73-metrics-client-ca\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.212899 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.212869 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-accelerators-collector-config\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.214397 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.214378 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-tls\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.214471 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.214433 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a106d45c-6d50-4df5-ae35-fce07bd9bf73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.219442 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.219423 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fshbl\" (UniqueName: \"kubernetes.io/projected/a106d45c-6d50-4df5-ae35-fce07bd9bf73-kube-api-access-fshbl\") pod \"node-exporter-wn7mt\" (UID: \"a106d45c-6d50-4df5-ae35-fce07bd9bf73\") " pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.354641 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:38.354604 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wn7mt" Jul 28 00:47:38.392753 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:38.392729 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda106d45c_6d50_4df5_ae35_fce07bd9bf73.slice/crio-54824fd5459df9494d37a6f1487291773d0821b8b00173c816d715eec41f190b WatchSource:0}: Error finding container 54824fd5459df9494d37a6f1487291773d0821b8b00173c816d715eec41f190b: Status 404 returned error can't find the container with id 54824fd5459df9494d37a6f1487291773d0821b8b00173c816d715eec41f190b Jul 28 00:47:39.179164 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:39.179128 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wn7mt" event={"ID":"a106d45c-6d50-4df5-ae35-fce07bd9bf73","Type":"ContainerStarted","Data":"54824fd5459df9494d37a6f1487291773d0821b8b00173c816d715eec41f190b"} Jul 28 00:47:40.076075 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.076047 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-777cb9787d-b2w6v"] Jul 28 00:47:40.080671 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.080643 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.083603 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.083459 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3jud4fvguh37e\"" Jul 28 00:47:40.083603 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.083476 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 00:47:40.083603 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.083493 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 00:47:40.083603 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.083510 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 00:47:40.083603 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.083520 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 00:47:40.083603 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.083544 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 00:47:40.083603 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.083548 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-24z5h\"" Jul 28 00:47:40.091513 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.091490 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-777cb9787d-b2w6v"] Jul 28 00:47:40.127896 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.127874 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.127993 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.127900 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzfq4\" (UniqueName: \"kubernetes.io/projected/25703e22-d2c7-40d3-8dcc-2478f30460ec-kube-api-access-xzfq4\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.127993 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.127928 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.127993 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.127981 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-tls\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.128138 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.128055 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-grpc-tls\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.128138 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.128074 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25703e22-d2c7-40d3-8dcc-2478f30460ec-metrics-client-ca\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.128138 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.128089 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.128138 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.128131 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.145497 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.145479 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qsxxt" Jul 28 00:47:40.185052 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.185028 2550 generic.go:356] "Generic (PLEG): container finished" podID="a106d45c-6d50-4df5-ae35-fce07bd9bf73" containerID="4febb96dc7774464650494e4c6114023097bbc581e4719600c68f0e3f7df5a01" exitCode=0 Jul 28 00:47:40.185407 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.185087 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wn7mt" event={"ID":"a106d45c-6d50-4df5-ae35-fce07bd9bf73","Type":"ContainerDied","Data":"4febb96dc7774464650494e4c6114023097bbc581e4719600c68f0e3f7df5a01"} Jul 28 00:47:40.229189 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.229162 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25703e22-d2c7-40d3-8dcc-2478f30460ec-metrics-client-ca\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.229279 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.229191 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.229279 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.229229 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.229387 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.229335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.229387 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.229365 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzfq4\" (UniqueName: \"kubernetes.io/projected/25703e22-d2c7-40d3-8dcc-2478f30460ec-kube-api-access-xzfq4\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.229483 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.229404 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.229483 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.229471 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-tls\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.229672 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.229570 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-grpc-tls\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.230309 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.230166 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25703e22-d2c7-40d3-8dcc-2478f30460ec-metrics-client-ca\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.232377 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.232351 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.232477 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.232389 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.232540 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.232507 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-grpc-tls\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.232818 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.232795 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.232903 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.232850 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.232980 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.232962 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/25703e22-d2c7-40d3-8dcc-2478f30460ec-secret-thanos-querier-tls\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.238484 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.238465 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzfq4\" (UniqueName: \"kubernetes.io/projected/25703e22-d2c7-40d3-8dcc-2478f30460ec-kube-api-access-xzfq4\") pod \"thanos-querier-777cb9787d-b2w6v\" (UID: \"25703e22-d2c7-40d3-8dcc-2478f30460ec\") " pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.389305 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.389244 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:40.509903 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:40.509877 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-777cb9787d-b2w6v"] Jul 28 00:47:40.512065 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:40.512035 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25703e22_d2c7_40d3_8dcc_2478f30460ec.slice/crio-6242b1b432b6923e44f66fac8cf3003e747a613be761d17de150f975cc524e5c WatchSource:0}: Error finding container 6242b1b432b6923e44f66fac8cf3003e747a613be761d17de150f975cc524e5c: Status 404 returned error can't find the container with id 6242b1b432b6923e44f66fac8cf3003e747a613be761d17de150f975cc524e5c Jul 28 00:47:41.191211 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:41.191163 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wn7mt" event={"ID":"a106d45c-6d50-4df5-ae35-fce07bd9bf73","Type":"ContainerStarted","Data":"abe96c4511c31ca44cd96e46057bf32bec33ab2f250b32d7de644a0f21d92194"} Jul 28 00:47:41.191685 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:41.191234 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wn7mt" event={"ID":"a106d45c-6d50-4df5-ae35-fce07bd9bf73","Type":"ContainerStarted","Data":"8d3374068781daaeb7e325a968ca10b81ab1b2999b19b08b7860af9d624a07f3"} Jul 28 00:47:41.192505 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:41.192483 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" event={"ID":"25703e22-d2c7-40d3-8dcc-2478f30460ec","Type":"ContainerStarted","Data":"6242b1b432b6923e44f66fac8cf3003e747a613be761d17de150f975cc524e5c"} Jul 28 00:47:42.436955 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.436924 2550 patch_prober.go:28] interesting pod/image-registry-559f46ddb8-kkksq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 28 00:47:42.437297 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.436986 2550 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" podUID="82f9b919-9e2e-4a87-9049-4691045afa53" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 00:47:42.787815 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.787752 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wn7mt" podStartSLOduration=3.799759418 podStartE2EDuration="4.787733045s" totalImagesPullingTime="922.675027ms" totalInitContainerRuntime="65.2986ms" isStatefulPod=true podCreationTimestamp="2026-07-28 00:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:41.210031526 +0000 UTC m=+56.970879537" watchObservedRunningTime="2026-07-28 00:47:42.787733045 +0000 UTC m=+58.548581059" Jul 28 00:47:42.788364 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.788341 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r"] Jul 28 00:47:42.790399 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.790382 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:42.792590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.792571 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hwhtx\"" Jul 28 00:47:42.792707 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.792588 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 00:47:42.802757 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.802693 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r"] Jul 28 00:47:42.851643 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.851614 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llxvp\" (UniqueName: \"kubernetes.io/projected/0f8733cf-36a4-49d4-b534-b13e4e6b0d4c-kube-api-access-llxvp\") pod \"monitoring-plugin-68b7fc9574-x2d5r\" (UID: \"0f8733cf-36a4-49d4-b534-b13e4e6b0d4c\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:42.851763 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.851669 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f8733cf-36a4-49d4-b534-b13e4e6b0d4c-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-x2d5r\" (UID: \"0f8733cf-36a4-49d4-b534-b13e4e6b0d4c\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:42.952066 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.952039 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llxvp\" (UniqueName: \"kubernetes.io/projected/0f8733cf-36a4-49d4-b534-b13e4e6b0d4c-kube-api-access-llxvp\") pod \"monitoring-plugin-68b7fc9574-x2d5r\" (UID: \"0f8733cf-36a4-49d4-b534-b13e4e6b0d4c\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:42.952156 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.952092 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f8733cf-36a4-49d4-b534-b13e4e6b0d4c-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-x2d5r\" (UID: \"0f8733cf-36a4-49d4-b534-b13e4e6b0d4c\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:42.952270 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:42.952250 2550 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 28 00:47:42.952322 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:47:42.952312 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f8733cf-36a4-49d4-b534-b13e4e6b0d4c-monitoring-plugin-cert podName:0f8733cf-36a4-49d4-b534-b13e4e6b0d4c nodeName:}" failed. No retries permitted until 2026-07-28 00:47:43.452295242 +0000 UTC m=+59.213143234 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0f8733cf-36a4-49d4-b534-b13e4e6b0d4c-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-x2d5r" (UID: "0f8733cf-36a4-49d4-b534-b13e4e6b0d4c") : secret "monitoring-plugin-cert" not found Jul 28 00:47:42.960773 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:42.960750 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llxvp\" (UniqueName: \"kubernetes.io/projected/0f8733cf-36a4-49d4-b534-b13e4e6b0d4c-kube-api-access-llxvp\") pod \"monitoring-plugin-68b7fc9574-x2d5r\" (UID: \"0f8733cf-36a4-49d4-b534-b13e4e6b0d4c\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:43.201607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:43.201538 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" event={"ID":"25703e22-d2c7-40d3-8dcc-2478f30460ec","Type":"ContainerStarted","Data":"93eebab50b93271c0ffce7933892f4cd01058e0c03c75f4fc1ef37b1f44538e3"} Jul 28 00:47:43.201607 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:43.201598 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" event={"ID":"25703e22-d2c7-40d3-8dcc-2478f30460ec","Type":"ContainerStarted","Data":"b80a530072cfbcbebe4fa80f64fadeafc75a7a79a956a8e83c05cfb5c4e0ddd4"} Jul 28 00:47:43.201811 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:43.201625 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" event={"ID":"25703e22-d2c7-40d3-8dcc-2478f30460ec","Type":"ContainerStarted","Data":"39dc3f670d37b61f71bc823a0b9107168b42f611914b35a2185a9876f71b560b"} Jul 28 00:47:43.456520 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:43.456453 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f8733cf-36a4-49d4-b534-b13e4e6b0d4c-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-x2d5r\" (UID: \"0f8733cf-36a4-49d4-b534-b13e4e6b0d4c\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:43.458776 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:43.458731 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f8733cf-36a4-49d4-b534-b13e4e6b0d4c-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-x2d5r\" (UID: \"0f8733cf-36a4-49d4-b534-b13e4e6b0d4c\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:43.698559 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:43.698519 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:43.824442 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:43.824412 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r"] Jul 28 00:47:43.827417 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:43.827390 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8733cf_36a4_49d4_b534_b13e4e6b0d4c.slice/crio-011119b52497c5b151ab3936df888c94f046791fa4721465ff5e7459ed08db42 WatchSource:0}: Error finding container 011119b52497c5b151ab3936df888c94f046791fa4721465ff5e7459ed08db42: Status 404 returned error can't find the container with id 011119b52497c5b151ab3936df888c94f046791fa4721465ff5e7459ed08db42 Jul 28 00:47:44.124083 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:44.124052 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:47:44.212624 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:44.212587 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" event={"ID":"25703e22-d2c7-40d3-8dcc-2478f30460ec","Type":"ContainerStarted","Data":"849f59599868477aff3bf538f27c483c2fb36336ec47cff4e7aa0c812f83a3e1"} Jul 28 00:47:44.212798 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:44.212666 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" event={"ID":"25703e22-d2c7-40d3-8dcc-2478f30460ec","Type":"ContainerStarted","Data":"53308c8af083d52a715c6215ac158da16eb37ca05e1f18a0be7485b8eef5a7c9"} Jul 28 00:47:44.212798 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:44.212709 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" event={"ID":"25703e22-d2c7-40d3-8dcc-2478f30460ec","Type":"ContainerStarted","Data":"696a5c9b05f30dff1d7bf421071d00e21ed0626ddf46440aff61bdedbdb9c6de"} Jul 28 00:47:44.212798 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:44.212759 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:44.213938 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:44.213907 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" event={"ID":"0f8733cf-36a4-49d4-b534-b13e4e6b0d4c","Type":"ContainerStarted","Data":"011119b52497c5b151ab3936df888c94f046791fa4721465ff5e7459ed08db42"} Jul 28 00:47:44.247970 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:44.247926 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" podStartSLOduration=1.626076544 podStartE2EDuration="4.247907645s" totalImagesPullingTime="2.621831101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:44.247120955 +0000 UTC m=+60.007968979" watchObservedRunningTime="2026-07-28 00:47:44.247907645 +0000 UTC m=+60.008755675" Jul 28 00:47:45.096559 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:45.096529 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nmbmm" Jul 28 00:47:46.228215 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:46.228171 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" event={"ID":"0f8733cf-36a4-49d4-b534-b13e4e6b0d4c","Type":"ContainerStarted","Data":"f05b2c6da68abed74f221020f88eb4c47bc804af8be502a06ae6fb2d134094a3"} Jul 28 00:47:46.228708 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:46.228430 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:46.236123 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:46.236097 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" Jul 28 00:47:46.245446 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:46.245397 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-x2d5r" podStartSLOduration=2.538493024 podStartE2EDuration="4.245382491s" totalImagesPullingTime="1.706889467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:46.24292215 +0000 UTC m=+62.003770162" watchObservedRunningTime="2026-07-28 00:47:46.245382491 +0000 UTC m=+62.006230505" Jul 28 00:47:46.714203 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:46.714171 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-559f46ddb8-kkksq"] Jul 28 00:47:50.224684 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.224654 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-777cb9787d-b2w6v" Jul 28 00:47:50.513690 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.513602 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:50.513690 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.513659 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:50.515976 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.515955 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 00:47:50.516040 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.515986 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 00:47:50.526736 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.526708 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7-original-pull-secret\") pod \"global-pull-secret-syncer-79x2c\" (UID: \"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7\") " pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:50.526869 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.526847 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07a0f217-710e-482b-a1bb-a358656d435f-metrics-certs\") pod \"network-metrics-daemon-gddjn\" (UID: \"07a0f217-710e-482b-a1bb-a358656d435f\") " pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:50.613950 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.613924 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:50.616406 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.616389 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 00:47:50.627220 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.627198 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 00:47:50.638002 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.637971 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbjvp\" (UniqueName: \"kubernetes.io/projected/57227b22-d127-4005-90b6-cc2d78d2fd38-kube-api-access-nbjvp\") pod \"network-check-target-8jqzf\" (UID: \"57227b22-d127-4005-90b6-cc2d78d2fd38\") " pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:50.688024 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.687986 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79x2c" Jul 28 00:47:50.799081 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.799059 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kdcl9\"" Jul 28 00:47:50.807184 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.807165 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-gddjn" Jul 28 00:47:50.807637 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.807615 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mkfth\"" Jul 28 00:47:50.808730 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.808710 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-79x2c"] Jul 28 00:47:50.811893 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:50.811871 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8abbb14_25bb_42e2_9fb5_0ad1c4f6ecd7.slice/crio-99ea43639037e6c450056a1cde8936292d9b902fa3fb7ba6cd756f4afaa60a04 WatchSource:0}: Error finding container 99ea43639037e6c450056a1cde8936292d9b902fa3fb7ba6cd756f4afaa60a04: Status 404 returned error can't find the container with id 99ea43639037e6c450056a1cde8936292d9b902fa3fb7ba6cd756f4afaa60a04 Jul 28 00:47:50.815569 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.815548 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:50.936712 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.936549 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jqzf"] Jul 28 00:47:50.939297 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:50.939271 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57227b22_d127_4005_90b6_cc2d78d2fd38.slice/crio-1db71c2070742bcbfe1a6b38841ac91900ad55b2269441b2bc5f402777eab303 WatchSource:0}: Error finding container 1db71c2070742bcbfe1a6b38841ac91900ad55b2269441b2bc5f402777eab303: Status 404 returned error can't find the container with id 1db71c2070742bcbfe1a6b38841ac91900ad55b2269441b2bc5f402777eab303 Jul 28 00:47:50.954724 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:50.954701 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-gddjn"] Jul 28 00:47:50.957521 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:47:50.957498 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a0f217_710e_482b_a1bb_a358656d435f.slice/crio-4f26a242fee7f4cbc201c73586f9ebc74bc283a1f0b36e8863be147bdd01d258 WatchSource:0}: Error finding container 4f26a242fee7f4cbc201c73586f9ebc74bc283a1f0b36e8863be147bdd01d258: Status 404 returned error can't find the container with id 4f26a242fee7f4cbc201c73586f9ebc74bc283a1f0b36e8863be147bdd01d258 Jul 28 00:47:51.248918 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:51.248836 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jqzf" event={"ID":"57227b22-d127-4005-90b6-cc2d78d2fd38","Type":"ContainerStarted","Data":"1db71c2070742bcbfe1a6b38841ac91900ad55b2269441b2bc5f402777eab303"} Jul 28 00:47:51.250233 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:51.250204 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gddjn" event={"ID":"07a0f217-710e-482b-a1bb-a358656d435f","Type":"ContainerStarted","Data":"4f26a242fee7f4cbc201c73586f9ebc74bc283a1f0b36e8863be147bdd01d258"} Jul 28 00:47:51.251447 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:51.251415 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-79x2c" event={"ID":"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7","Type":"ContainerStarted","Data":"99ea43639037e6c450056a1cde8936292d9b902fa3fb7ba6cd756f4afaa60a04"} Jul 28 00:47:53.279873 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:53.279403 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gddjn" event={"ID":"07a0f217-710e-482b-a1bb-a358656d435f","Type":"ContainerStarted","Data":"ab1f8fa5b70fe73b0b708384a510ba925baeb4e2c806e5ce3ba42f1bc5685fdd"} Jul 28 00:47:53.279873 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:53.279469 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-gddjn" event={"ID":"07a0f217-710e-482b-a1bb-a358656d435f","Type":"ContainerStarted","Data":"4ee84147778a581ba5f2b264b9a3950fc6af042c24c863492e090ed31ef9cacd"} Jul 28 00:47:53.298163 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:53.298098 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-gddjn" podStartSLOduration=66.847605933 podStartE2EDuration="1m8.298081588s" totalImagesPullingTime="1.450475655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:53.297070544 +0000 UTC m=+69.057918560" watchObservedRunningTime="2026-07-28 00:47:53.298081588 +0000 UTC m=+69.058929603" Jul 28 00:47:56.293647 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:56.293606 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jqzf" event={"ID":"57227b22-d127-4005-90b6-cc2d78d2fd38","Type":"ContainerStarted","Data":"b7a3b87942b2c0a2f85bd3b6fe36624ca96bc69dcd1e1eefcd0e79728fa1e98d"} Jul 28 00:47:56.294247 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:56.293694 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:47:56.295448 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:56.295419 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-79x2c" event={"ID":"a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7","Type":"ContainerStarted","Data":"8c6d0b9beef0031419f766be6ad495b0f51809491026c9498507005f658f0710"} Jul 28 00:47:56.307439 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:56.307387 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8jqzf" podStartSLOduration=66.788564639 podStartE2EDuration="1m11.307373416s" totalImagesPullingTime="4.518808777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:56.30717521 +0000 UTC m=+72.068023224" watchObservedRunningTime="2026-07-28 00:47:56.307373416 +0000 UTC m=+72.068221429" Jul 28 00:47:56.319941 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:47:56.319899 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-79x2c" podStartSLOduration=66.672149879 podStartE2EDuration="1m11.319886338s" totalImagesPullingTime="4.647736459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:56.319560731 +0000 UTC m=+72.080408744" watchObservedRunningTime="2026-07-28 00:47:56.319886338 +0000 UTC m=+72.080734434" Jul 28 00:48:11.732497 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:11.732372 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" podUID="82f9b919-9e2e-4a87-9049-4691045afa53" containerName="registry" containerID="cri-o://f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0" gracePeriod=30 Jul 28 00:48:11.981551 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:11.981530 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:48:12.071760 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.071730 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82f9b919-9e2e-4a87-9049-4691045afa53-ca-trust-extracted\") pod \"82f9b919-9e2e-4a87-9049-4691045afa53\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " Jul 28 00:48:12.071903 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.071774 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-registry-certificates\") pod \"82f9b919-9e2e-4a87-9049-4691045afa53\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " Jul 28 00:48:12.071903 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.071809 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-registry-tls\") pod \"82f9b919-9e2e-4a87-9049-4691045afa53\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " Jul 28 00:48:12.072021 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.071936 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-bound-sa-token\") pod \"82f9b919-9e2e-4a87-9049-4691045afa53\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " Jul 28 00:48:12.072021 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.071984 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpstk\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-kube-api-access-zpstk\") pod \"82f9b919-9e2e-4a87-9049-4691045afa53\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " Jul 28 00:48:12.072137 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.072039 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-trusted-ca\") pod \"82f9b919-9e2e-4a87-9049-4691045afa53\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " Jul 28 00:48:12.072137 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.072087 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-installation-pull-secrets\") pod \"82f9b919-9e2e-4a87-9049-4691045afa53\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " Jul 28 00:48:12.072137 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.072119 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-image-registry-private-configuration\") pod \"82f9b919-9e2e-4a87-9049-4691045afa53\" (UID: \"82f9b919-9e2e-4a87-9049-4691045afa53\") " Jul 28 00:48:12.072757 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.072293 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "82f9b919-9e2e-4a87-9049-4691045afa53" (UID: "82f9b919-9e2e-4a87-9049-4691045afa53"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:48:12.072757 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.072706 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "82f9b919-9e2e-4a87-9049-4691045afa53" (UID: "82f9b919-9e2e-4a87-9049-4691045afa53"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:48:12.074687 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.074630 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-kube-api-access-zpstk" (OuterVolumeSpecName: "kube-api-access-zpstk") pod "82f9b919-9e2e-4a87-9049-4691045afa53" (UID: "82f9b919-9e2e-4a87-9049-4691045afa53"). InnerVolumeSpecName "kube-api-access-zpstk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:48:12.074792 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.074683 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "82f9b919-9e2e-4a87-9049-4691045afa53" (UID: "82f9b919-9e2e-4a87-9049-4691045afa53"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:48:12.074792 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.074749 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "82f9b919-9e2e-4a87-9049-4691045afa53" (UID: "82f9b919-9e2e-4a87-9049-4691045afa53"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:48:12.074869 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.074802 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "82f9b919-9e2e-4a87-9049-4691045afa53" (UID: "82f9b919-9e2e-4a87-9049-4691045afa53"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:48:12.074869 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.074827 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "82f9b919-9e2e-4a87-9049-4691045afa53" (UID: "82f9b919-9e2e-4a87-9049-4691045afa53"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:48:12.081776 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.081750 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f9b919-9e2e-4a87-9049-4691045afa53-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "82f9b919-9e2e-4a87-9049-4691045afa53" (UID: "82f9b919-9e2e-4a87-9049-4691045afa53"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:48:12.172983 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.172960 2550 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-installation-pull-secrets\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:48:12.172983 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.172981 2550 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/82f9b919-9e2e-4a87-9049-4691045afa53-image-registry-private-configuration\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:48:12.173114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.172991 2550 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/82f9b919-9e2e-4a87-9049-4691045afa53-ca-trust-extracted\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:48:12.173114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.173000 2550 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-registry-certificates\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:48:12.173114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.173028 2550 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-registry-tls\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:48:12.173114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.173037 2550 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-bound-sa-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:48:12.173114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.173044 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpstk\" (UniqueName: \"kubernetes.io/projected/82f9b919-9e2e-4a87-9049-4691045afa53-kube-api-access-zpstk\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:48:12.173114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.173052 2550 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/82f9b919-9e2e-4a87-9049-4691045afa53-trusted-ca\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:48:12.356426 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.356353 2550 generic.go:356] "Generic (PLEG): container finished" podID="82f9b919-9e2e-4a87-9049-4691045afa53" containerID="f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0" exitCode=0 Jul 28 00:48:12.356426 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.356401 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" Jul 28 00:48:12.356426 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.356407 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" event={"ID":"82f9b919-9e2e-4a87-9049-4691045afa53","Type":"ContainerDied","Data":"f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0"} Jul 28 00:48:12.356601 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.356456 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-559f46ddb8-kkksq" event={"ID":"82f9b919-9e2e-4a87-9049-4691045afa53","Type":"ContainerDied","Data":"6b079d9534a62dafdb5ebe389ba1a83021fae5dfcb4b2aa560be6ca7ba22f9f5"} Jul 28 00:48:12.356601 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.356482 2550 scope.go:117] "RemoveContainer" containerID="f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0" Jul 28 00:48:12.365761 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.365741 2550 scope.go:117] "RemoveContainer" containerID="f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0" Jul 28 00:48:12.366000 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:48:12.365978 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0\": container with ID starting with f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0 not found: ID does not exist" containerID="f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0" Jul 28 00:48:12.366072 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.366022 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0"} err="failed to get container status \"f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0\": rpc error: code = NotFound desc = could not find container \"f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0\": container with ID starting with f3d486d2d0b990e6ea1288fa56247dbd94829128b20cf20bc3d4ba78a5c826c0 not found: ID does not exist" Jul 28 00:48:12.376084 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.376063 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-559f46ddb8-kkksq"] Jul 28 00:48:12.379049 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.379029 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-559f46ddb8-kkksq"] Jul 28 00:48:12.782806 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:12.782725 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82f9b919-9e2e-4a87-9049-4691045afa53" path="/var/lib/kubelet/pods/82f9b919-9e2e-4a87-9049-4691045afa53/volumes" Jul 28 00:48:27.301583 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:27.301546 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8jqzf" Jul 28 00:48:59.862100 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.862058 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 00:48:59.862811 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.862358 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82f9b919-9e2e-4a87-9049-4691045afa53" containerName="registry" Jul 28 00:48:59.862811 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.862372 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="82f9b919-9e2e-4a87-9049-4691045afa53" containerName="registry" Jul 28 00:48:59.862811 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.862421 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="82f9b919-9e2e-4a87-9049-4691045afa53" containerName="registry" Jul 28 00:48:59.869800 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.869770 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.871917 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.871884 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 00:48:59.871917 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.871899 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 00:48:59.872167 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.871900 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 00:48:59.872167 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.872045 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 00:48:59.872264 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.872217 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 00:48:59.872314 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.872274 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 00:48:59.872461 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.872413 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hnbvj\"" Jul 28 00:48:59.872461 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.872449 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 00:48:59.872564 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.872496 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 00:48:59.877869 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.877843 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 00:48:59.878367 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.878117 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 00:48:59.883635 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883607 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f81656e6-540d-4544-8dbf-ce0693bc08e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883737 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883642 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883737 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883663 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883737 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883689 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883737 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883732 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81656e6-540d-4544-8dbf-ce0693bc08e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883971 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883767 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccrr\" (UniqueName: \"kubernetes.io/projected/f81656e6-540d-4544-8dbf-ce0693bc08e1-kube-api-access-gccrr\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883971 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883836 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f81656e6-540d-4544-8dbf-ce0693bc08e1-config-out\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883971 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883870 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883971 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883908 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883971 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883926 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-web-config\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883971 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883946 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f81656e6-540d-4544-8dbf-ce0693bc08e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.883971 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883968 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f81656e6-540d-4544-8dbf-ce0693bc08e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.884402 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.883984 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.984778 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984751 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.984778 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984779 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-web-config\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.984953 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984799 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f81656e6-540d-4544-8dbf-ce0693bc08e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.984953 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984816 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f81656e6-540d-4544-8dbf-ce0693bc08e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.984953 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984839 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.984953 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984878 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f81656e6-540d-4544-8dbf-ce0693bc08e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.984953 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984910 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.984953 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984940 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.985273 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984963 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.985273 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.984988 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81656e6-540d-4544-8dbf-ce0693bc08e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.985273 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.985037 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccrr\" (UniqueName: \"kubernetes.io/projected/f81656e6-540d-4544-8dbf-ce0693bc08e1-kube-api-access-gccrr\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.985273 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.985076 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f81656e6-540d-4544-8dbf-ce0693bc08e1-config-out\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.985273 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.985106 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.985273 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.985266 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f81656e6-540d-4544-8dbf-ce0693bc08e1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.986131 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.985765 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f81656e6-540d-4544-8dbf-ce0693bc08e1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.987002 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.986939 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f81656e6-540d-4544-8dbf-ce0693bc08e1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.987990 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.987930 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.988121 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.988040 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-config-volume\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.988309 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.988288 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.988376 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.988293 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.988417 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.988404 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-web-config\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.989149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.989128 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.989386 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.989366 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f81656e6-540d-4544-8dbf-ce0693bc08e1-config-out\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.989718 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.989701 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f81656e6-540d-4544-8dbf-ce0693bc08e1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.989930 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.989911 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f81656e6-540d-4544-8dbf-ce0693bc08e1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:48:59.992615 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:48:59.992592 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccrr\" (UniqueName: \"kubernetes.io/projected/f81656e6-540d-4544-8dbf-ce0693bc08e1-kube-api-access-gccrr\") pod \"alertmanager-main-0\" (UID: \"f81656e6-540d-4544-8dbf-ce0693bc08e1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:49:00.180775 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:00.180722 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 00:49:00.306887 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:00.306862 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 00:49:00.309020 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:49:00.308981 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf81656e6_540d_4544_8dbf_ce0693bc08e1.slice/crio-6bf3f3e454b1d0ac73b1f8a95f288833098fda0df137d9db5454b72b6913c8e2 WatchSource:0}: Error finding container 6bf3f3e454b1d0ac73b1f8a95f288833098fda0df137d9db5454b72b6913c8e2: Status 404 returned error can't find the container with id 6bf3f3e454b1d0ac73b1f8a95f288833098fda0df137d9db5454b72b6913c8e2 Jul 28 00:49:00.534430 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:00.534359 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f81656e6-540d-4544-8dbf-ce0693bc08e1","Type":"ContainerStarted","Data":"6bf3f3e454b1d0ac73b1f8a95f288833098fda0df137d9db5454b72b6913c8e2"} Jul 28 00:49:01.539622 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:01.539593 2550 generic.go:356] "Generic (PLEG): container finished" podID="f81656e6-540d-4544-8dbf-ce0693bc08e1" containerID="699bdc9a22c4e250a70133094380c7c79c203c46ee9f59ff8f9ca350752514c6" exitCode=0 Jul 28 00:49:01.539900 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:01.539663 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f81656e6-540d-4544-8dbf-ce0693bc08e1","Type":"ContainerDied","Data":"699bdc9a22c4e250a70133094380c7c79c203c46ee9f59ff8f9ca350752514c6"} Jul 28 00:49:02.304393 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.304358 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-684d6665f8-7qh58"] Jul 28 00:49:02.307100 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.307032 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.309585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.309438 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 00:49:02.309585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.309576 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 00:49:02.309816 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.309794 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ntjm2\"" Jul 28 00:49:02.309816 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.309794 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 00:49:02.309983 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.309874 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 00:49:02.310189 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.310175 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 00:49:02.322721 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.322693 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 00:49:02.323132 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.323035 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-684d6665f8-7qh58"] Jul 28 00:49:02.401611 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.401583 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-telemeter-trusted-ca-bundle\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.401611 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.401613 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-serving-certs-ca-bundle\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.401758 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.401630 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-telemeter-client-tls\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.401758 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.401648 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bkqt\" (UniqueName: \"kubernetes.io/projected/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-kube-api-access-7bkqt\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.401758 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.401743 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-federate-client-tls\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.401862 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.401770 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-metrics-client-ca\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.401862 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.401795 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-secret-telemeter-client\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.401862 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.401849 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.503364 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.503292 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.503541 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.503467 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-telemeter-trusted-ca-bundle\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.503612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.503560 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-serving-certs-ca-bundle\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.503731 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.503713 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-telemeter-client-tls\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.503845 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.503794 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bkqt\" (UniqueName: \"kubernetes.io/projected/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-kube-api-access-7bkqt\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.503984 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.503966 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-federate-client-tls\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.504056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.503997 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-metrics-client-ca\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.504118 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.504056 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-secret-telemeter-client\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.504423 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.504399 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-telemeter-trusted-ca-bundle\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.504922 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.504875 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-serving-certs-ca-bundle\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.505043 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.504973 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-metrics-client-ca\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.506123 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.506097 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.507560 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.507432 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-telemeter-client-tls\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.507944 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.507920 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-federate-client-tls\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.508055 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.507923 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-secret-telemeter-client\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.514543 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.514519 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bkqt\" (UniqueName: \"kubernetes.io/projected/bb08d8bb-6c62-47e4-9c7c-826f5bc65920-kube-api-access-7bkqt\") pod \"telemeter-client-684d6665f8-7qh58\" (UID: \"bb08d8bb-6c62-47e4-9c7c-826f5bc65920\") " pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.621571 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.621490 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" Jul 28 00:49:02.783103 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:02.783081 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-684d6665f8-7qh58"] Jul 28 00:49:03.092985 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:49:03.092952 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb08d8bb_6c62_47e4_9c7c_826f5bc65920.slice/crio-7864d6baa2ac9090bf677c5eeb2615f2f18119d74703ca60e86c35ca2840115a WatchSource:0}: Error finding container 7864d6baa2ac9090bf677c5eeb2615f2f18119d74703ca60e86c35ca2840115a: Status 404 returned error can't find the container with id 7864d6baa2ac9090bf677c5eeb2615f2f18119d74703ca60e86c35ca2840115a Jul 28 00:49:03.550561 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:03.550531 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f81656e6-540d-4544-8dbf-ce0693bc08e1","Type":"ContainerStarted","Data":"4f6feca1c1b04304f569ea67c6d09a714da8b2a787c63a388a6c810341c6b5ea"} Jul 28 00:49:03.550679 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:03.550604 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f81656e6-540d-4544-8dbf-ce0693bc08e1","Type":"ContainerStarted","Data":"8f22ea88443acfeb2ee88796b40c62776307b10aefe6263a3db0bc6d9930b348"} Jul 28 00:49:03.550679 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:03.550647 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f81656e6-540d-4544-8dbf-ce0693bc08e1","Type":"ContainerStarted","Data":"1c65264824310ed4391962b65168d35713531e8d70a1909b20b1b1d97d059a0e"} Jul 28 00:49:03.550793 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:03.550684 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f81656e6-540d-4544-8dbf-ce0693bc08e1","Type":"ContainerStarted","Data":"16e1970e2a105f516f35ef610c57fef77a59e0cd258c3f3508489aedbd319926"} Jul 28 00:49:03.550793 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:03.550727 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f81656e6-540d-4544-8dbf-ce0693bc08e1","Type":"ContainerStarted","Data":"91bac266d32e2712950e921780e840f14878012859cf2974524d34d98e84c40b"} Jul 28 00:49:03.550793 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:03.550769 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f81656e6-540d-4544-8dbf-ce0693bc08e1","Type":"ContainerStarted","Data":"b24ac5c6c045f49e59660500291f814f834d940bf30a590c9ff3188d45127ae2"} Jul 28 00:49:03.551625 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:03.551606 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" event={"ID":"bb08d8bb-6c62-47e4-9c7c-826f5bc65920","Type":"ContainerStarted","Data":"7864d6baa2ac9090bf677c5eeb2615f2f18119d74703ca60e86c35ca2840115a"} Jul 28 00:49:03.575016 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:03.574973 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.93726306 podStartE2EDuration="4.574958286s" totalImagesPullingTime="2.62398728s" totalInitContainerRuntime="13.707946ms" isStatefulPod=false podCreationTimestamp="2026-07-28 00:48:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:49:03.573051009 +0000 UTC m=+139.333899027" watchObservedRunningTime="2026-07-28 00:49:03.574958286 +0000 UTC m=+139.335806296" Jul 28 00:49:05.565000 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:05.564957 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" event={"ID":"bb08d8bb-6c62-47e4-9c7c-826f5bc65920","Type":"ContainerStarted","Data":"3420b2575c0250171c4dd21aaa9e6192c41a15c0454d2f8dde623266dac910ac"} Jul 28 00:49:05.565601 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:05.565057 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" event={"ID":"bb08d8bb-6c62-47e4-9c7c-826f5bc65920","Type":"ContainerStarted","Data":"4e938bcce3877271164bc892d7459a98889c4eff672c8bc5d2699fc3743ab07a"} Jul 28 00:49:05.565601 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:05.565096 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" event={"ID":"bb08d8bb-6c62-47e4-9c7c-826f5bc65920","Type":"ContainerStarted","Data":"a91b610038bbb033952fc3ec3c0e46f09b72e65c58caea326ead2f8d9c4d9573"} Jul 28 00:49:05.589675 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:49:05.589624 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-684d6665f8-7qh58" podStartSLOduration=1.6998297359999999 podStartE2EDuration="3.589609517s" totalImagesPullingTime="1.889779781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:49:05.588504373 +0000 UTC m=+141.349352385" watchObservedRunningTime="2026-07-28 00:49:05.589609517 +0000 UTC m=+141.350457529" Jul 28 00:51:19.056956 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.056867 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm"] Jul 28 00:51:19.060178 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.060158 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.062730 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.062706 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6kfwp\"" Jul 28 00:51:19.062843 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.062706 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 00:51:19.063555 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.063535 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 00:51:19.070257 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.070234 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm"] Jul 28 00:51:19.123620 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.123588 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.123708 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.123621 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2952\" (UniqueName: \"kubernetes.io/projected/3bc6217c-acb4-455f-9a04-09707fd32411-kube-api-access-c2952\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.123708 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.123645 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.224623 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.224595 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.224709 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.224632 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2952\" (UniqueName: \"kubernetes.io/projected/3bc6217c-acb4-455f-9a04-09707fd32411-kube-api-access-c2952\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.224709 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.224653 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.225018 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.224990 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.225064 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.225028 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.233264 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.233241 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2952\" (UniqueName: \"kubernetes.io/projected/3bc6217c-acb4-455f-9a04-09707fd32411-kube-api-access-c2952\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.372379 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.372331 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:19.496123 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:19.496097 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm"] Jul 28 00:51:19.497255 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:51:19.497227 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc6217c_acb4_455f_9a04_09707fd32411.slice/crio-07e4ebad01ec2d2ca2d9c0085e8bef200c4b79a486d72e7e27d0e6590df4f3cc WatchSource:0}: Error finding container 07e4ebad01ec2d2ca2d9c0085e8bef200c4b79a486d72e7e27d0e6590df4f3cc: Status 404 returned error can't find the container with id 07e4ebad01ec2d2ca2d9c0085e8bef200c4b79a486d72e7e27d0e6590df4f3cc Jul 28 00:51:20.054497 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:20.054459 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" event={"ID":"3bc6217c-acb4-455f-9a04-09707fd32411","Type":"ContainerStarted","Data":"07e4ebad01ec2d2ca2d9c0085e8bef200c4b79a486d72e7e27d0e6590df4f3cc"} Jul 28 00:51:26.078852 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:26.078821 2550 generic.go:356] "Generic (PLEG): container finished" podID="3bc6217c-acb4-455f-9a04-09707fd32411" containerID="9cd2e80ff68c64f667dea873844e788f9aecbb7b189a999723366869c1d7c12d" exitCode=0 Jul 28 00:51:26.079194 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:26.078889 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" event={"ID":"3bc6217c-acb4-455f-9a04-09707fd32411","Type":"ContainerDied","Data":"9cd2e80ff68c64f667dea873844e788f9aecbb7b189a999723366869c1d7c12d"} Jul 28 00:51:28.088827 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:28.088796 2550 generic.go:356] "Generic (PLEG): container finished" podID="3bc6217c-acb4-455f-9a04-09707fd32411" containerID="cea9beadfb60c8fd2b4dfe0db1ebcd66bba204d8f4f87ea698544b1863abcfcc" exitCode=0 Jul 28 00:51:28.089168 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:28.088875 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" event={"ID":"3bc6217c-acb4-455f-9a04-09707fd32411","Type":"ContainerDied","Data":"cea9beadfb60c8fd2b4dfe0db1ebcd66bba204d8f4f87ea698544b1863abcfcc"} Jul 28 00:51:36.120778 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:36.120743 2550 generic.go:356] "Generic (PLEG): container finished" podID="3bc6217c-acb4-455f-9a04-09707fd32411" containerID="6e7da49904c2b28ee6daf6aaa9db8e30dd15851382492b2688fa680c017a61f7" exitCode=0 Jul 28 00:51:36.121162 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:36.120798 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" event={"ID":"3bc6217c-acb4-455f-9a04-09707fd32411","Type":"ContainerDied","Data":"6e7da49904c2b28ee6daf6aaa9db8e30dd15851382492b2688fa680c017a61f7"} Jul 28 00:51:37.253831 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.253810 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:37.354623 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.354599 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2952\" (UniqueName: \"kubernetes.io/projected/3bc6217c-acb4-455f-9a04-09707fd32411-kube-api-access-c2952\") pod \"3bc6217c-acb4-455f-9a04-09707fd32411\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " Jul 28 00:51:37.354753 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.354632 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-bundle\") pod \"3bc6217c-acb4-455f-9a04-09707fd32411\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " Jul 28 00:51:37.354753 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.354676 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-util\") pod \"3bc6217c-acb4-455f-9a04-09707fd32411\" (UID: \"3bc6217c-acb4-455f-9a04-09707fd32411\") " Jul 28 00:51:37.355322 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.355296 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-bundle" (OuterVolumeSpecName: "bundle") pod "3bc6217c-acb4-455f-9a04-09707fd32411" (UID: "3bc6217c-acb4-455f-9a04-09707fd32411"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:51:37.356732 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.356703 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc6217c-acb4-455f-9a04-09707fd32411-kube-api-access-c2952" (OuterVolumeSpecName: "kube-api-access-c2952") pod "3bc6217c-acb4-455f-9a04-09707fd32411" (UID: "3bc6217c-acb4-455f-9a04-09707fd32411"). InnerVolumeSpecName "kube-api-access-c2952". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:51:37.359282 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.359263 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-util" (OuterVolumeSpecName: "util") pod "3bc6217c-acb4-455f-9a04-09707fd32411" (UID: "3bc6217c-acb4-455f-9a04-09707fd32411"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:51:37.455109 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.455059 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-util\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:51:37.455109 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.455078 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2952\" (UniqueName: \"kubernetes.io/projected/3bc6217c-acb4-455f-9a04-09707fd32411-kube-api-access-c2952\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:51:37.455109 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:37.455088 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3bc6217c-acb4-455f-9a04-09707fd32411-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:51:38.130217 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:38.130188 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" Jul 28 00:51:38.130385 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:38.130182 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540jkbcm" event={"ID":"3bc6217c-acb4-455f-9a04-09707fd32411","Type":"ContainerDied","Data":"07e4ebad01ec2d2ca2d9c0085e8bef200c4b79a486d72e7e27d0e6590df4f3cc"} Jul 28 00:51:38.130385 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:38.130313 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07e4ebad01ec2d2ca2d9c0085e8bef200c4b79a486d72e7e27d0e6590df4f3cc" Jul 28 00:51:41.978224 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:41.978193 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk"] Jul 28 00:51:41.978612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:41.978476 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc6217c-acb4-455f-9a04-09707fd32411" containerName="pull" Jul 28 00:51:41.978612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:41.978490 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc6217c-acb4-455f-9a04-09707fd32411" containerName="pull" Jul 28 00:51:41.978612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:41.978514 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc6217c-acb4-455f-9a04-09707fd32411" containerName="extract" Jul 28 00:51:41.978612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:41.978521 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc6217c-acb4-455f-9a04-09707fd32411" containerName="extract" Jul 28 00:51:41.978612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:41.978528 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bc6217c-acb4-455f-9a04-09707fd32411" containerName="util" Jul 28 00:51:41.978612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:41.978534 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bc6217c-acb4-455f-9a04-09707fd32411" containerName="util" Jul 28 00:51:41.978612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:41.978584 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bc6217c-acb4-455f-9a04-09707fd32411" containerName="extract" Jul 28 00:51:42.019681 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.019656 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk"] Jul 28 00:51:42.019830 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.019741 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" Jul 28 00:51:42.022386 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.022351 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-5p82w\"" Jul 28 00:51:42.022386 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.022373 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 00:51:42.022551 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.022354 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 00:51:42.087823 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.087801 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3fb62b2-3e0e-46cc-900a-ea95bb321d5f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x6tvk\" (UID: \"d3fb62b2-3e0e-46cc-900a-ea95bb321d5f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" Jul 28 00:51:42.087952 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.087835 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vppm\" (UniqueName: \"kubernetes.io/projected/d3fb62b2-3e0e-46cc-900a-ea95bb321d5f-kube-api-access-8vppm\") pod \"cert-manager-operator-controller-manager-55c444bf86-x6tvk\" (UID: \"d3fb62b2-3e0e-46cc-900a-ea95bb321d5f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" Jul 28 00:51:42.188801 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.188779 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3fb62b2-3e0e-46cc-900a-ea95bb321d5f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x6tvk\" (UID: \"d3fb62b2-3e0e-46cc-900a-ea95bb321d5f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" Jul 28 00:51:42.188903 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.188812 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vppm\" (UniqueName: \"kubernetes.io/projected/d3fb62b2-3e0e-46cc-900a-ea95bb321d5f-kube-api-access-8vppm\") pod \"cert-manager-operator-controller-manager-55c444bf86-x6tvk\" (UID: \"d3fb62b2-3e0e-46cc-900a-ea95bb321d5f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" Jul 28 00:51:42.189229 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.189211 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3fb62b2-3e0e-46cc-900a-ea95bb321d5f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-x6tvk\" (UID: \"d3fb62b2-3e0e-46cc-900a-ea95bb321d5f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" Jul 28 00:51:42.196968 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.196950 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vppm\" (UniqueName: \"kubernetes.io/projected/d3fb62b2-3e0e-46cc-900a-ea95bb321d5f-kube-api-access-8vppm\") pod \"cert-manager-operator-controller-manager-55c444bf86-x6tvk\" (UID: \"d3fb62b2-3e0e-46cc-900a-ea95bb321d5f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" Jul 28 00:51:42.328719 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.328696 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" Jul 28 00:51:42.475520 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:42.475491 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk"] Jul 28 00:51:42.477989 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:51:42.477964 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fb62b2_3e0e_46cc_900a_ea95bb321d5f.slice/crio-bd81d6dc8c320a740342f429933310b67b3495e9a0bd5c518e53a65462639b69 WatchSource:0}: Error finding container bd81d6dc8c320a740342f429933310b67b3495e9a0bd5c518e53a65462639b69: Status 404 returned error can't find the container with id bd81d6dc8c320a740342f429933310b67b3495e9a0bd5c518e53a65462639b69 Jul 28 00:51:43.150403 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:43.150364 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" event={"ID":"d3fb62b2-3e0e-46cc-900a-ea95bb321d5f","Type":"ContainerStarted","Data":"bd81d6dc8c320a740342f429933310b67b3495e9a0bd5c518e53a65462639b69"} Jul 28 00:51:44.607535 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:44.607511 2550 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 00:51:45.158841 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:45.158804 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" event={"ID":"d3fb62b2-3e0e-46cc-900a-ea95bb321d5f","Type":"ContainerStarted","Data":"632ac31272a12e9550326b0520bae8c57f1d4b21c92c1cef54527178d98a3ac3"} Jul 28 00:51:45.246448 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:45.179481 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-x6tvk" podStartSLOduration=2.229442642 podStartE2EDuration="4.179466507s" totalImagesPullingTime="1.950023865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:51:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:51:45.177921706 +0000 UTC m=+300.938769719" watchObservedRunningTime="2026-07-28 00:51:45.179466507 +0000 UTC m=+300.940314520" Jul 28 00:51:46.281727 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.281693 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq"] Jul 28 00:51:46.295342 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.295317 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.299288 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.299224 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 00:51:46.299424 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.299250 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6kfwp\"" Jul 28 00:51:46.299671 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.299549 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 00:51:46.301444 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.301417 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq"] Jul 28 00:51:46.418356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.418330 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.418455 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.418387 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.418455 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.418408 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwfp\" (UniqueName: \"kubernetes.io/projected/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-kube-api-access-6dwfp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.519533 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.519505 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.519624 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.519589 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.519683 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.519619 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwfp\" (UniqueName: \"kubernetes.io/projected/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-kube-api-access-6dwfp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.519948 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.519924 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.520032 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.519986 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.527548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.527524 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwfp\" (UniqueName: \"kubernetes.io/projected/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-kube-api-access-6dwfp\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.605313 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.605295 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:46.742720 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:46.742692 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq"] Jul 28 00:51:46.745098 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:51:46.745070 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec42b09_fa35_4e3f_8ede_fa5992ab1b7d.slice/crio-0efb959341da04f616a05dd5270261c5b2c8b389c4dd373f387b01217495fe52 WatchSource:0}: Error finding container 0efb959341da04f616a05dd5270261c5b2c8b389c4dd373f387b01217495fe52: Status 404 returned error can't find the container with id 0efb959341da04f616a05dd5270261c5b2c8b389c4dd373f387b01217495fe52 Jul 28 00:51:47.167618 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:47.167553 2550 generic.go:356] "Generic (PLEG): container finished" podID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerID="29f442f3ed15f1d12d19dbef0b9b442f361241c203688c4f114743d66d407423" exitCode=0 Jul 28 00:51:47.167618 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:47.167616 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" event={"ID":"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d","Type":"ContainerDied","Data":"29f442f3ed15f1d12d19dbef0b9b442f361241c203688c4f114743d66d407423"} Jul 28 00:51:47.167800 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:47.167650 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" event={"ID":"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d","Type":"ContainerStarted","Data":"0efb959341da04f616a05dd5270261c5b2c8b389c4dd373f387b01217495fe52"} Jul 28 00:51:47.168554 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:47.168538 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:51:48.228620 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.228587 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hm42x"] Jul 28 00:51:48.245372 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.245342 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hm42x"] Jul 28 00:51:48.245508 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.245444 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" Jul 28 00:51:48.247876 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.247852 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 00:51:48.248681 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.248632 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 00:51:48.248681 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.248659 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nx22w\"" Jul 28 00:51:48.333088 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.333060 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f7a565c-49a9-4cba-88c7-620abcdb2fc0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hm42x\" (UID: \"3f7a565c-49a9-4cba-88c7-620abcdb2fc0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" Jul 28 00:51:48.333205 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.333100 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6lnf\" (UniqueName: \"kubernetes.io/projected/3f7a565c-49a9-4cba-88c7-620abcdb2fc0-kube-api-access-r6lnf\") pod \"cert-manager-cainjector-559cdd8c87-hm42x\" (UID: \"3f7a565c-49a9-4cba-88c7-620abcdb2fc0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" Jul 28 00:51:48.433984 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.433955 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f7a565c-49a9-4cba-88c7-620abcdb2fc0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hm42x\" (UID: \"3f7a565c-49a9-4cba-88c7-620abcdb2fc0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" Jul 28 00:51:48.434160 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.433996 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6lnf\" (UniqueName: \"kubernetes.io/projected/3f7a565c-49a9-4cba-88c7-620abcdb2fc0-kube-api-access-r6lnf\") pod \"cert-manager-cainjector-559cdd8c87-hm42x\" (UID: \"3f7a565c-49a9-4cba-88c7-620abcdb2fc0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" Jul 28 00:51:48.442780 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.442760 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f7a565c-49a9-4cba-88c7-620abcdb2fc0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hm42x\" (UID: \"3f7a565c-49a9-4cba-88c7-620abcdb2fc0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" Jul 28 00:51:48.443055 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.443031 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6lnf\" (UniqueName: \"kubernetes.io/projected/3f7a565c-49a9-4cba-88c7-620abcdb2fc0-kube-api-access-r6lnf\") pod \"cert-manager-cainjector-559cdd8c87-hm42x\" (UID: \"3f7a565c-49a9-4cba-88c7-620abcdb2fc0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" Jul 28 00:51:48.564106 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:48.564076 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" Jul 28 00:51:49.150098 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:49.150077 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hm42x"] Jul 28 00:51:49.152461 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:51:49.152424 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f7a565c_49a9_4cba_88c7_620abcdb2fc0.slice/crio-8a37b0878b2b1efe2cc35f8abbeef944dee7385dc7160cf5542f4883f7670af3 WatchSource:0}: Error finding container 8a37b0878b2b1efe2cc35f8abbeef944dee7385dc7160cf5542f4883f7670af3: Status 404 returned error can't find the container with id 8a37b0878b2b1efe2cc35f8abbeef944dee7385dc7160cf5542f4883f7670af3 Jul 28 00:51:49.177902 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:49.177875 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" event={"ID":"3f7a565c-49a9-4cba-88c7-620abcdb2fc0","Type":"ContainerStarted","Data":"8a37b0878b2b1efe2cc35f8abbeef944dee7385dc7160cf5542f4883f7670af3"} Jul 28 00:51:49.179700 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:49.179663 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" event={"ID":"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d","Type":"ContainerStarted","Data":"6978303b607ef8146b037fae18e0ed110292d14d93608dcd7e64064534a307d2"} Jul 28 00:51:50.185129 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:50.185098 2550 generic.go:356] "Generic (PLEG): container finished" podID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerID="6978303b607ef8146b037fae18e0ed110292d14d93608dcd7e64064534a307d2" exitCode=0 Jul 28 00:51:50.185632 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:50.185172 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" event={"ID":"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d","Type":"ContainerDied","Data":"6978303b607ef8146b037fae18e0ed110292d14d93608dcd7e64064534a307d2"} Jul 28 00:51:51.192360 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:51.192324 2550 generic.go:356] "Generic (PLEG): container finished" podID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerID="c0d4a47021c6755c2c5897d69d53a505601b89c6cc40d97321e2a40f8d297c37" exitCode=0 Jul 28 00:51:51.192775 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:51.192369 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" event={"ID":"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d","Type":"ContainerDied","Data":"c0d4a47021c6755c2c5897d69d53a505601b89c6cc40d97321e2a40f8d297c37"} Jul 28 00:51:52.198312 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.198271 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" event={"ID":"3f7a565c-49a9-4cba-88c7-620abcdb2fc0","Type":"ContainerStarted","Data":"9723c571ed85098555ea503df4e588dcf185ded044122d5bc627172a605bb474"} Jul 28 00:51:52.214002 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.213915 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hm42x" podStartSLOduration=1.414412816 podStartE2EDuration="4.213899873s" totalImagesPullingTime="2.799487057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:51:52.212954324 +0000 UTC m=+307.973802337" watchObservedRunningTime="2026-07-28 00:51:52.213899873 +0000 UTC m=+307.974747885" Jul 28 00:51:52.339379 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.339356 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:52.465560 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.465496 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dwfp\" (UniqueName: \"kubernetes.io/projected/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-kube-api-access-6dwfp\") pod \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " Jul 28 00:51:52.465560 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.465551 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-bundle\") pod \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " Jul 28 00:51:52.465751 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.465570 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-util\") pod \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\" (UID: \"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d\") " Jul 28 00:51:52.465916 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.465892 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-bundle" (OuterVolumeSpecName: "bundle") pod "fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" (UID: "fec42b09-fa35-4e3f-8ede-fa5992ab1b7d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:51:52.467490 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.467470 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-kube-api-access-6dwfp" (OuterVolumeSpecName: "kube-api-access-6dwfp") pod "fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" (UID: "fec42b09-fa35-4e3f-8ede-fa5992ab1b7d"). InnerVolumeSpecName "kube-api-access-6dwfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:51:52.469986 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.469959 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-util" (OuterVolumeSpecName: "util") pod "fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" (UID: "fec42b09-fa35-4e3f-8ede-fa5992ab1b7d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:51:52.566475 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.566442 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-util\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:51:52.566475 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.566470 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwfp\" (UniqueName: \"kubernetes.io/projected/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-kube-api-access-6dwfp\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:51:52.566643 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:52.566484 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fec42b09-fa35-4e3f-8ede-fa5992ab1b7d-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:51:53.204774 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:53.204708 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" Jul 28 00:51:53.204774 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:53.204713 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fgbsgq" event={"ID":"fec42b09-fa35-4e3f-8ede-fa5992ab1b7d","Type":"ContainerDied","Data":"0efb959341da04f616a05dd5270261c5b2c8b389c4dd373f387b01217495fe52"} Jul 28 00:51:53.204774 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:53.204762 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0efb959341da04f616a05dd5270261c5b2c8b389c4dd373f387b01217495fe52" Jul 28 00:51:56.823394 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.823354 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qdwhq"] Jul 28 00:51:56.823797 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.823646 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerName="util" Jul 28 00:51:56.823797 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.823657 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerName="util" Jul 28 00:51:56.823797 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.823673 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerName="pull" Jul 28 00:51:56.823797 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.823680 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerName="pull" Jul 28 00:51:56.823797 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.823690 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerName="extract" Jul 28 00:51:56.823797 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.823695 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerName="extract" Jul 28 00:51:56.823797 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.823745 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec42b09-fa35-4e3f-8ede-fa5992ab1b7d" containerName="extract" Jul 28 00:51:56.825755 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.825735 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qdwhq" Jul 28 00:51:56.828101 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.828066 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-kq4xm\"" Jul 28 00:51:56.835451 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.835426 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qdwhq"] Jul 28 00:51:56.897346 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.897317 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67lrv\" (UniqueName: \"kubernetes.io/projected/f33806ab-db1c-46d8-bf35-a73704a340fd-kube-api-access-67lrv\") pod \"cert-manager-55b4d954d5-qdwhq\" (UID: \"f33806ab-db1c-46d8-bf35-a73704a340fd\") " pod="cert-manager/cert-manager-55b4d954d5-qdwhq" Jul 28 00:51:56.897455 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.897359 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f33806ab-db1c-46d8-bf35-a73704a340fd-bound-sa-token\") pod \"cert-manager-55b4d954d5-qdwhq\" (UID: \"f33806ab-db1c-46d8-bf35-a73704a340fd\") " pod="cert-manager/cert-manager-55b4d954d5-qdwhq" Jul 28 00:51:56.998638 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.998611 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67lrv\" (UniqueName: \"kubernetes.io/projected/f33806ab-db1c-46d8-bf35-a73704a340fd-kube-api-access-67lrv\") pod \"cert-manager-55b4d954d5-qdwhq\" (UID: \"f33806ab-db1c-46d8-bf35-a73704a340fd\") " pod="cert-manager/cert-manager-55b4d954d5-qdwhq" Jul 28 00:51:56.998729 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:56.998651 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f33806ab-db1c-46d8-bf35-a73704a340fd-bound-sa-token\") pod \"cert-manager-55b4d954d5-qdwhq\" (UID: \"f33806ab-db1c-46d8-bf35-a73704a340fd\") " pod="cert-manager/cert-manager-55b4d954d5-qdwhq" Jul 28 00:51:57.010734 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:57.010709 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f33806ab-db1c-46d8-bf35-a73704a340fd-bound-sa-token\") pod \"cert-manager-55b4d954d5-qdwhq\" (UID: \"f33806ab-db1c-46d8-bf35-a73704a340fd\") " pod="cert-manager/cert-manager-55b4d954d5-qdwhq" Jul 28 00:51:57.016990 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:57.016967 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67lrv\" (UniqueName: \"kubernetes.io/projected/f33806ab-db1c-46d8-bf35-a73704a340fd-kube-api-access-67lrv\") pod \"cert-manager-55b4d954d5-qdwhq\" (UID: \"f33806ab-db1c-46d8-bf35-a73704a340fd\") " pod="cert-manager/cert-manager-55b4d954d5-qdwhq" Jul 28 00:51:57.136353 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:57.136287 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qdwhq" Jul 28 00:51:57.263319 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:57.263289 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qdwhq"] Jul 28 00:51:57.266158 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:51:57.266134 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf33806ab_db1c_46d8_bf35_a73704a340fd.slice/crio-4947466e87a3f77c23d3a7798bfcf83f7af96d76f79c8df774b2c2eeb29f0a71 WatchSource:0}: Error finding container 4947466e87a3f77c23d3a7798bfcf83f7af96d76f79c8df774b2c2eeb29f0a71: Status 404 returned error can't find the container with id 4947466e87a3f77c23d3a7798bfcf83f7af96d76f79c8df774b2c2eeb29f0a71 Jul 28 00:51:58.227140 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:58.227101 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qdwhq" event={"ID":"f33806ab-db1c-46d8-bf35-a73704a340fd","Type":"ContainerStarted","Data":"3fd12cc2c9f13c6b7256f28fd41fc372334e4c771fe5eec67d1802c241531d07"} Jul 28 00:51:58.227503 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:58.227148 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qdwhq" event={"ID":"f33806ab-db1c-46d8-bf35-a73704a340fd","Type":"ContainerStarted","Data":"4947466e87a3f77c23d3a7798bfcf83f7af96d76f79c8df774b2c2eeb29f0a71"} Jul 28 00:51:58.242907 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:58.242843 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-qdwhq" podStartSLOduration=2.242826897 podStartE2EDuration="2.242826897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:51:58.242514776 +0000 UTC m=+314.003362790" watchObservedRunningTime="2026-07-28 00:51:58.242826897 +0000 UTC m=+314.003674910" Jul 28 00:51:59.866197 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:59.866161 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b"] Jul 28 00:51:59.868335 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:59.868315 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" Jul 28 00:51:59.872436 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:59.872406 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-6nmv6\"" Jul 28 00:51:59.872578 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:59.872439 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 00:51:59.873067 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:59.873047 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 00:51:59.887824 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:59.887795 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b"] Jul 28 00:51:59.916642 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:59.916611 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945b8226-b92e-4d96-bcaa-806cd25ca8b1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4st2b\" (UID: \"945b8226-b92e-4d96-bcaa-806cd25ca8b1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" Jul 28 00:51:59.916762 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:51:59.916687 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhvj\" (UniqueName: \"kubernetes.io/projected/945b8226-b92e-4d96-bcaa-806cd25ca8b1-kube-api-access-fjhvj\") pod \"openshift-lws-operator-fd8ccff4c-4st2b\" (UID: \"945b8226-b92e-4d96-bcaa-806cd25ca8b1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" Jul 28 00:52:00.017893 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:00.017869 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhvj\" (UniqueName: \"kubernetes.io/projected/945b8226-b92e-4d96-bcaa-806cd25ca8b1-kube-api-access-fjhvj\") pod \"openshift-lws-operator-fd8ccff4c-4st2b\" (UID: \"945b8226-b92e-4d96-bcaa-806cd25ca8b1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" Jul 28 00:52:00.017982 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:00.017906 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945b8226-b92e-4d96-bcaa-806cd25ca8b1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4st2b\" (UID: \"945b8226-b92e-4d96-bcaa-806cd25ca8b1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" Jul 28 00:52:00.018218 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:00.018200 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/945b8226-b92e-4d96-bcaa-806cd25ca8b1-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4st2b\" (UID: \"945b8226-b92e-4d96-bcaa-806cd25ca8b1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" Jul 28 00:52:00.026866 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:00.026847 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhvj\" (UniqueName: \"kubernetes.io/projected/945b8226-b92e-4d96-bcaa-806cd25ca8b1-kube-api-access-fjhvj\") pod \"openshift-lws-operator-fd8ccff4c-4st2b\" (UID: \"945b8226-b92e-4d96-bcaa-806cd25ca8b1\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" Jul 28 00:52:00.178278 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:00.178214 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" Jul 28 00:52:00.307801 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:00.307776 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b"] Jul 28 00:52:00.309560 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:52:00.309533 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod945b8226_b92e_4d96_bcaa_806cd25ca8b1.slice/crio-f9fe99882f79188b4418da106d334cf49a7064d8e7e57b7603f5b5de48856bf9 WatchSource:0}: Error finding container f9fe99882f79188b4418da106d334cf49a7064d8e7e57b7603f5b5de48856bf9: Status 404 returned error can't find the container with id f9fe99882f79188b4418da106d334cf49a7064d8e7e57b7603f5b5de48856bf9 Jul 28 00:52:01.244121 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:01.244083 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" event={"ID":"945b8226-b92e-4d96-bcaa-806cd25ca8b1","Type":"ContainerStarted","Data":"f9fe99882f79188b4418da106d334cf49a7064d8e7e57b7603f5b5de48856bf9"} Jul 28 00:52:02.249104 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:02.249035 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" event={"ID":"945b8226-b92e-4d96-bcaa-806cd25ca8b1","Type":"ContainerStarted","Data":"2738b68919bff05d1f6f8dcd6eb3496863a56d1c1feacc3eb41f28bee8d78976"} Jul 28 00:52:02.266426 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:02.266376 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4st2b" podStartSLOduration=1.763428988 podStartE2EDuration="3.266358552s" totalImagesPullingTime="1.502929564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:02.265137811 +0000 UTC m=+318.025985824" watchObservedRunningTime="2026-07-28 00:52:02.266358552 +0000 UTC m=+318.027206566" Jul 28 00:52:04.977201 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:04.977158 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v"] Jul 28 00:52:04.979569 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:04.979553 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:04.982372 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:04.982348 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6kfwp\"" Jul 28 00:52:04.982772 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:04.982755 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 00:52:04.983222 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:04.983204 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 00:52:04.989292 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:04.989270 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v"] Jul 28 00:52:05.056316 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.056292 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56r7g\" (UniqueName: \"kubernetes.io/projected/3dddd224-b570-47b5-9cba-3caf33f5be97-kube-api-access-56r7g\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.056404 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.056333 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.056404 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.056356 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.157548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.157527 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.157647 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.157560 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.157715 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.157693 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56r7g\" (UniqueName: \"kubernetes.io/projected/3dddd224-b570-47b5-9cba-3caf33f5be97-kube-api-access-56r7g\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.157894 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.157878 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.158027 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.157987 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.165457 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.165436 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56r7g\" (UniqueName: \"kubernetes.io/projected/3dddd224-b570-47b5-9cba-3caf33f5be97-kube-api-access-56r7g\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.288578 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.288551 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:05.410862 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:05.410710 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v"] Jul 28 00:52:05.413433 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:52:05.413410 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dddd224_b570_47b5_9cba_3caf33f5be97.slice/crio-64f3ed7850c8ce0f93d1aa93395733bfac3adf97ccf78c6eb116a9644aea722c WatchSource:0}: Error finding container 64f3ed7850c8ce0f93d1aa93395733bfac3adf97ccf78c6eb116a9644aea722c: Status 404 returned error can't find the container with id 64f3ed7850c8ce0f93d1aa93395733bfac3adf97ccf78c6eb116a9644aea722c Jul 28 00:52:06.267802 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:06.267768 2550 generic.go:356] "Generic (PLEG): container finished" podID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerID="97083e1b2eb8fc5095c8b7b1727cbda69a0c1c9b30d04954f37153ff3db21746" exitCode=0 Jul 28 00:52:06.268187 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:06.267864 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" event={"ID":"3dddd224-b570-47b5-9cba-3caf33f5be97","Type":"ContainerDied","Data":"97083e1b2eb8fc5095c8b7b1727cbda69a0c1c9b30d04954f37153ff3db21746"} Jul 28 00:52:06.268187 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:06.267928 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" event={"ID":"3dddd224-b570-47b5-9cba-3caf33f5be97","Type":"ContainerStarted","Data":"64f3ed7850c8ce0f93d1aa93395733bfac3adf97ccf78c6eb116a9644aea722c"} Jul 28 00:52:07.280895 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:07.280820 2550 generic.go:356] "Generic (PLEG): container finished" podID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerID="32baa1c7eb6e8b63b4d693d7f7c101f106902b5957d591dc8f73f21ccb404a8f" exitCode=0 Jul 28 00:52:07.281247 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:07.280901 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" event={"ID":"3dddd224-b570-47b5-9cba-3caf33f5be97","Type":"ContainerDied","Data":"32baa1c7eb6e8b63b4d693d7f7c101f106902b5957d591dc8f73f21ccb404a8f"} Jul 28 00:52:08.289166 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:08.289125 2550 generic.go:356] "Generic (PLEG): container finished" podID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerID="278fd79f981b554ecd2cc02aeba07aa0493ecd0671cc6815434fee2362c27cca" exitCode=0 Jul 28 00:52:08.289562 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:08.289182 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" event={"ID":"3dddd224-b570-47b5-9cba-3caf33f5be97","Type":"ContainerDied","Data":"278fd79f981b554ecd2cc02aeba07aa0493ecd0671cc6815434fee2362c27cca"} Jul 28 00:52:09.415830 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.415808 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:09.489294 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.489268 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-util\") pod \"3dddd224-b570-47b5-9cba-3caf33f5be97\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " Jul 28 00:52:09.489408 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.489321 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56r7g\" (UniqueName: \"kubernetes.io/projected/3dddd224-b570-47b5-9cba-3caf33f5be97-kube-api-access-56r7g\") pod \"3dddd224-b570-47b5-9cba-3caf33f5be97\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " Jul 28 00:52:09.489408 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.489357 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-bundle\") pod \"3dddd224-b570-47b5-9cba-3caf33f5be97\" (UID: \"3dddd224-b570-47b5-9cba-3caf33f5be97\") " Jul 28 00:52:09.490144 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.490119 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-bundle" (OuterVolumeSpecName: "bundle") pod "3dddd224-b570-47b5-9cba-3caf33f5be97" (UID: "3dddd224-b570-47b5-9cba-3caf33f5be97"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:52:09.491266 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.491237 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dddd224-b570-47b5-9cba-3caf33f5be97-kube-api-access-56r7g" (OuterVolumeSpecName: "kube-api-access-56r7g") pod "3dddd224-b570-47b5-9cba-3caf33f5be97" (UID: "3dddd224-b570-47b5-9cba-3caf33f5be97"). InnerVolumeSpecName "kube-api-access-56r7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:52:09.495096 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.495073 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-util" (OuterVolumeSpecName: "util") pod "3dddd224-b570-47b5-9cba-3caf33f5be97" (UID: "3dddd224-b570-47b5-9cba-3caf33f5be97"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:52:09.590536 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.590482 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-util\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:52:09.590536 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.590503 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56r7g\" (UniqueName: \"kubernetes.io/projected/3dddd224-b570-47b5-9cba-3caf33f5be97-kube-api-access-56r7g\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:52:09.590536 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:09.590514 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3dddd224-b570-47b5-9cba-3caf33f5be97-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:52:10.301749 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:10.301713 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" event={"ID":"3dddd224-b570-47b5-9cba-3caf33f5be97","Type":"ContainerDied","Data":"64f3ed7850c8ce0f93d1aa93395733bfac3adf97ccf78c6eb116a9644aea722c"} Jul 28 00:52:10.301749 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:10.301746 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afthp6v" Jul 28 00:52:10.301932 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:10.301773 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64f3ed7850c8ce0f93d1aa93395733bfac3adf97ccf78c6eb116a9644aea722c" Jul 28 00:52:23.002848 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.002808 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h"] Jul 28 00:52:23.003356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.003128 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerName="util" Jul 28 00:52:23.003356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.003140 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerName="util" Jul 28 00:52:23.003356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.003157 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerName="extract" Jul 28 00:52:23.003356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.003163 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerName="extract" Jul 28 00:52:23.003356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.003171 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerName="pull" Jul 28 00:52:23.003356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.003177 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerName="pull" Jul 28 00:52:23.003356 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.003232 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dddd224-b570-47b5-9cba-3caf33f5be97" containerName="extract" Jul 28 00:52:23.009921 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.009898 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.013250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.013215 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 00:52:23.013411 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.013273 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 00:52:23.013411 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.013281 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 00:52:23.013411 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.013325 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 00:52:23.013563 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.013481 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-s5btk\"" Jul 28 00:52:23.026952 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.026927 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h"] Jul 28 00:52:23.083279 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.083248 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4gkj\" (UniqueName: \"kubernetes.io/projected/3e35e475-16c0-49a6-9825-4422336ea1ac-kube-api-access-n4gkj\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.083400 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.083295 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e35e475-16c0-49a6-9825-4422336ea1ac-tmp\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.083400 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.083368 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e35e475-16c0-49a6-9825-4422336ea1ac-manifests\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.083400 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.083394 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e35e475-16c0-49a6-9825-4422336ea1ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.083508 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.083431 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e35e475-16c0-49a6-9825-4422336ea1ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.184614 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.184590 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4gkj\" (UniqueName: \"kubernetes.io/projected/3e35e475-16c0-49a6-9825-4422336ea1ac-kube-api-access-n4gkj\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.184721 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.184625 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e35e475-16c0-49a6-9825-4422336ea1ac-tmp\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.184721 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.184648 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e35e475-16c0-49a6-9825-4422336ea1ac-manifests\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.184721 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.184666 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e35e475-16c0-49a6-9825-4422336ea1ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.184721 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.184693 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e35e475-16c0-49a6-9825-4422336ea1ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.185080 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.185052 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3e35e475-16c0-49a6-9825-4422336ea1ac-manifests\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.186864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.186837 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3e35e475-16c0-49a6-9825-4422336ea1ac-tmp\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.187099 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.187081 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3e35e475-16c0-49a6-9825-4422336ea1ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.187146 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.187087 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3e35e475-16c0-49a6-9825-4422336ea1ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.202124 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.202099 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4gkj\" (UniqueName: \"kubernetes.io/projected/3e35e475-16c0-49a6-9825-4422336ea1ac-kube-api-access-n4gkj\") pod \"opendatahub-operator-controller-manager-644f76477d-dmn5h\" (UID: \"3e35e475-16c0-49a6-9825-4422336ea1ac\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.320286 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.320261 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:23.472135 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:23.471184 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h"] Jul 28 00:52:24.366233 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:24.366187 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" event={"ID":"3e35e475-16c0-49a6-9825-4422336ea1ac","Type":"ContainerStarted","Data":"fc9700a5b5829a590ccd337989e75c5fc942824e229daa606d68509eb50fcbbb"} Jul 28 00:52:27.381787 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:27.381754 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" event={"ID":"3e35e475-16c0-49a6-9825-4422336ea1ac","Type":"ContainerStarted","Data":"6f63121ce87551139f18a8ceefe6811269344b7742771643b5801455f85a2ea7"} Jul 28 00:52:28.388629 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:28.388595 2550 generic.go:356] "Generic (PLEG): container finished" podID="3e35e475-16c0-49a6-9825-4422336ea1ac" containerID="6f63121ce87551139f18a8ceefe6811269344b7742771643b5801455f85a2ea7" exitCode=0 Jul 28 00:52:28.389002 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:28.388653 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" event={"ID":"3e35e475-16c0-49a6-9825-4422336ea1ac","Type":"ContainerDied","Data":"6f63121ce87551139f18a8ceefe6811269344b7742771643b5801455f85a2ea7"} Jul 28 00:52:29.395408 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:29.395369 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" event={"ID":"3e35e475-16c0-49a6-9825-4422336ea1ac","Type":"ContainerStarted","Data":"beef3b2f1b2c444faed72859b6809c1d4f112005e063d95ded0b615e2d8eb18c"} Jul 28 00:52:29.395793 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:29.395453 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:29.417709 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:29.417647 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" podStartSLOduration=3.00443474 podStartE2EDuration="7.417627194s" totalImagesPullingTime="3.348797296s" totalInitContainerRuntime="1.064395158s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:29.416426748 +0000 UTC m=+345.177274761" watchObservedRunningTime="2026-07-28 00:52:29.417627194 +0000 UTC m=+345.178475209" Jul 28 00:52:40.402206 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:40.402122 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-dmn5h" Jul 28 00:52:42.836263 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:42.836230 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lfxrm"] Jul 28 00:52:42.840241 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:42.840223 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:42.842754 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:42.842721 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-j57d6\"" Jul 28 00:52:42.842891 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:42.842873 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 28 00:52:42.848823 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:42.848799 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lfxrm"] Jul 28 00:52:43.033369 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.033326 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5h9f\" (UniqueName: \"kubernetes.io/projected/8f186614-7ab6-493e-ae34-a00688b60446-kube-api-access-n5h9f\") pod \"odh-model-controller-7998c67dc4-lfxrm\" (UID: \"8f186614-7ab6-493e-ae34-a00688b60446\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:43.033487 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.033428 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f186614-7ab6-493e-ae34-a00688b60446-cert\") pod \"odh-model-controller-7998c67dc4-lfxrm\" (UID: \"8f186614-7ab6-493e-ae34-a00688b60446\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:43.134484 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.134420 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5h9f\" (UniqueName: \"kubernetes.io/projected/8f186614-7ab6-493e-ae34-a00688b60446-kube-api-access-n5h9f\") pod \"odh-model-controller-7998c67dc4-lfxrm\" (UID: \"8f186614-7ab6-493e-ae34-a00688b60446\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:43.134484 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.134476 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f186614-7ab6-493e-ae34-a00688b60446-cert\") pod \"odh-model-controller-7998c67dc4-lfxrm\" (UID: \"8f186614-7ab6-493e-ae34-a00688b60446\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:43.136637 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.136619 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8f186614-7ab6-493e-ae34-a00688b60446-cert\") pod \"odh-model-controller-7998c67dc4-lfxrm\" (UID: \"8f186614-7ab6-493e-ae34-a00688b60446\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:43.143051 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.143030 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5h9f\" (UniqueName: \"kubernetes.io/projected/8f186614-7ab6-493e-ae34-a00688b60446-kube-api-access-n5h9f\") pod \"odh-model-controller-7998c67dc4-lfxrm\" (UID: \"8f186614-7ab6-493e-ae34-a00688b60446\") " pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:43.151581 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.151564 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:43.276570 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.276550 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-lfxrm"] Jul 28 00:52:43.279073 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:52:43.279050 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f186614_7ab6_493e_ae34_a00688b60446.slice/crio-5331b674b99d3bea2b393501804f4504889702b98135c60bf4622615c06dd2cc WatchSource:0}: Error finding container 5331b674b99d3bea2b393501804f4504889702b98135c60bf4622615c06dd2cc: Status 404 returned error can't find the container with id 5331b674b99d3bea2b393501804f4504889702b98135c60bf4622615c06dd2cc Jul 28 00:52:43.458498 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.458434 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" event={"ID":"8f186614-7ab6-493e-ae34-a00688b60446","Type":"ContainerStarted","Data":"5331b674b99d3bea2b393501804f4504889702b98135c60bf4622615c06dd2cc"} Jul 28 00:52:43.526726 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.526698 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sq4gg"] Jul 28 00:52:43.531597 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.531581 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:43.533711 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.533695 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 28 00:52:43.537413 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.537392 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/788e61d4-30c3-47f3-918f-847e8b4eda03-tls-certs\") pod \"model-serving-api-54d89bd5fc-sq4gg\" (UID: \"788e61d4-30c3-47f3-918f-847e8b4eda03\") " pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:43.537507 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.537491 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s94xr\" (UniqueName: \"kubernetes.io/projected/788e61d4-30c3-47f3-918f-847e8b4eda03-kube-api-access-s94xr\") pod \"model-serving-api-54d89bd5fc-sq4gg\" (UID: \"788e61d4-30c3-47f3-918f-847e8b4eda03\") " pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:43.544724 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.544686 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sq4gg"] Jul 28 00:52:43.637945 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.637923 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s94xr\" (UniqueName: \"kubernetes.io/projected/788e61d4-30c3-47f3-918f-847e8b4eda03-kube-api-access-s94xr\") pod \"model-serving-api-54d89bd5fc-sq4gg\" (UID: \"788e61d4-30c3-47f3-918f-847e8b4eda03\") " pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:43.638047 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.637964 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/788e61d4-30c3-47f3-918f-847e8b4eda03-tls-certs\") pod \"model-serving-api-54d89bd5fc-sq4gg\" (UID: \"788e61d4-30c3-47f3-918f-847e8b4eda03\") " pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:43.638139 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:52:43.638123 2550 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 28 00:52:43.638191 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:52:43.638179 2550 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788e61d4-30c3-47f3-918f-847e8b4eda03-tls-certs podName:788e61d4-30c3-47f3-918f-847e8b4eda03 nodeName:}" failed. No retries permitted until 2026-07-28 00:52:44.138164095 +0000 UTC m=+359.899012086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/788e61d4-30c3-47f3-918f-847e8b4eda03-tls-certs") pod "model-serving-api-54d89bd5fc-sq4gg" (UID: "788e61d4-30c3-47f3-918f-847e8b4eda03") : secret "model-serving-api-tls" not found Jul 28 00:52:43.646633 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:43.646615 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s94xr\" (UniqueName: \"kubernetes.io/projected/788e61d4-30c3-47f3-918f-847e8b4eda03-kube-api-access-s94xr\") pod \"model-serving-api-54d89bd5fc-sq4gg\" (UID: \"788e61d4-30c3-47f3-918f-847e8b4eda03\") " pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:44.142907 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:44.142809 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/788e61d4-30c3-47f3-918f-847e8b4eda03-tls-certs\") pod \"model-serving-api-54d89bd5fc-sq4gg\" (UID: \"788e61d4-30c3-47f3-918f-847e8b4eda03\") " pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:44.146733 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:44.146703 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/788e61d4-30c3-47f3-918f-847e8b4eda03-tls-certs\") pod \"model-serving-api-54d89bd5fc-sq4gg\" (UID: \"788e61d4-30c3-47f3-918f-847e8b4eda03\") " pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:44.441536 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:44.441454 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:44.578563 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:44.578541 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sq4gg"] Jul 28 00:52:44.580985 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:52:44.580957 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788e61d4_30c3_47f3_918f_847e8b4eda03.slice/crio-4da49088a0bb280a2371386c0ff87943434b61e6d6dcc7f6dfdfc52169da63d7 WatchSource:0}: Error finding container 4da49088a0bb280a2371386c0ff87943434b61e6d6dcc7f6dfdfc52169da63d7: Status 404 returned error can't find the container with id 4da49088a0bb280a2371386c0ff87943434b61e6d6dcc7f6dfdfc52169da63d7 Jul 28 00:52:45.472056 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:45.471999 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" event={"ID":"788e61d4-30c3-47f3-918f-847e8b4eda03","Type":"ContainerStarted","Data":"4da49088a0bb280a2371386c0ff87943434b61e6d6dcc7f6dfdfc52169da63d7"} Jul 28 00:52:46.478518 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:46.478429 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" event={"ID":"8f186614-7ab6-493e-ae34-a00688b60446","Type":"ContainerStarted","Data":"0208fe3277c221c88c99ab906d4063a83251749899104e9a29300497137250e2"} Jul 28 00:52:46.478940 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:46.478540 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:46.494930 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:46.494885 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" podStartSLOduration=1.643901505 podStartE2EDuration="4.494873595s" totalImagesPullingTime="2.85097209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:46.493728759 +0000 UTC m=+362.254576776" watchObservedRunningTime="2026-07-28 00:52:46.494873595 +0000 UTC m=+362.255721607" Jul 28 00:52:47.483985 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:47.483953 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" event={"ID":"788e61d4-30c3-47f3-918f-847e8b4eda03","Type":"ContainerStarted","Data":"d79a8df88f4b53af04011af3a834662c1c1a723c1383ed8ed2877d6d28b3420a"} Jul 28 00:52:47.484317 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:47.484208 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:52:47.500452 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:47.500410 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" podStartSLOduration=1.7428506449999999 podStartE2EDuration="4.500396497s" totalImagesPullingTime="2.757545852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:47.498705222 +0000 UTC m=+363.259553237" watchObservedRunningTime="2026-07-28 00:52:47.500396497 +0000 UTC m=+363.261244510" Jul 28 00:52:49.298635 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.298603 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd"] Jul 28 00:52:49.302076 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.302055 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.305423 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.305377 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 00:52:49.305543 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.305438 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 00:52:49.305543 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.305455 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 00:52:49.305543 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.305472 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-24vfz\"" Jul 28 00:52:49.314458 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.314429 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd"] Jul 28 00:52:49.378349 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.378315 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/469d5192-e317-4924-805a-3154a8240376-manager-config\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.378349 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.378352 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/469d5192-e317-4924-805a-3154a8240376-metrics-cert\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.378575 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.378381 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/469d5192-e317-4924-805a-3154a8240376-cert\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.378575 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.378523 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rctlp\" (UniqueName: \"kubernetes.io/projected/469d5192-e317-4924-805a-3154a8240376-kube-api-access-rctlp\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.479811 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.479782 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/469d5192-e317-4924-805a-3154a8240376-manager-config\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.479811 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.479812 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/469d5192-e317-4924-805a-3154a8240376-metrics-cert\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.480033 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.479839 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/469d5192-e317-4924-805a-3154a8240376-cert\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.480033 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.479940 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rctlp\" (UniqueName: \"kubernetes.io/projected/469d5192-e317-4924-805a-3154a8240376-kube-api-access-rctlp\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.480484 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.480463 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/469d5192-e317-4924-805a-3154a8240376-manager-config\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.482322 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.482300 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/469d5192-e317-4924-805a-3154a8240376-metrics-cert\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.482416 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.482335 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/469d5192-e317-4924-805a-3154a8240376-cert\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.497926 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.497906 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rctlp\" (UniqueName: \"kubernetes.io/projected/469d5192-e317-4924-805a-3154a8240376-kube-api-access-rctlp\") pod \"lws-controller-manager-f79f7c78b-q2wxd\" (UID: \"469d5192-e317-4924-805a-3154a8240376\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.612768 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.612711 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:49.739613 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:49.739592 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd"] Jul 28 00:52:49.741611 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:52:49.741584 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod469d5192_e317_4924_805a_3154a8240376.slice/crio-f2ffe480edcda8f549ca22610b6985a0128f720da815a7a216233d08c1481e59 WatchSource:0}: Error finding container f2ffe480edcda8f549ca22610b6985a0128f720da815a7a216233d08c1481e59: Status 404 returned error can't find the container with id f2ffe480edcda8f549ca22610b6985a0128f720da815a7a216233d08c1481e59 Jul 28 00:52:50.500493 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:50.500454 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" event={"ID":"469d5192-e317-4924-805a-3154a8240376","Type":"ContainerStarted","Data":"f2ffe480edcda8f549ca22610b6985a0128f720da815a7a216233d08c1481e59"} Jul 28 00:52:51.506476 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:51.506379 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" event={"ID":"469d5192-e317-4924-805a-3154a8240376","Type":"ContainerStarted","Data":"b7acf36c7756f2008cd2d3e86f21c6d8876861cb26f7e6377a59eb2536710a4c"} Jul 28 00:52:51.506476 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:51.506468 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:52:51.526973 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:51.526927 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" podStartSLOduration=1.2116726309999999 podStartE2EDuration="2.526913827s" totalImagesPullingTime="1.315241196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:51.526288034 +0000 UTC m=+367.287136051" watchObservedRunningTime="2026-07-28 00:52:51.526913827 +0000 UTC m=+367.287761887" Jul 28 00:52:57.486554 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:57.486523 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-lfxrm" Jul 28 00:52:58.493572 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:52:58.493545 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-sq4gg" Jul 28 00:53:00.608491 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.608461 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n"] Jul 28 00:53:00.615117 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.615096 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.617994 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.617968 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 00:53:00.617994 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.617986 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4lq2b\"" Jul 28 00:53:00.618294 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.618277 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 00:53:00.618353 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.618339 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 00:53:00.624545 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.624522 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n"] Jul 28 00:53:00.663001 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.662979 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.663123 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.663034 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.663123 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.663052 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.663123 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.663069 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.663222 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.663141 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.663222 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.663176 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.663222 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.663201 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.663313 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.663223 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.663313 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.663254 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krkn9\" (UniqueName: \"kubernetes.io/projected/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-kube-api-access-krkn9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764293 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764270 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764384 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764303 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764384 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764319 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764560 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764541 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764595 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764572 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764631 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764605 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764681 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764632 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764681 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764661 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krkn9\" (UniqueName: \"kubernetes.io/projected/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-kube-api-access-krkn9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764776 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764697 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764776 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764748 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764776 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764752 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.764956 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.764932 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.765088 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.765066 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.765331 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.765308 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.766831 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.766811 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.767086 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.767068 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.773024 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.772991 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.773435 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.773412 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krkn9\" (UniqueName: \"kubernetes.io/projected/2034427d-6a25-41c3-9aeb-5e1d4c50f5ec-kube-api-access-krkn9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n\" (UID: \"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:00.926243 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:00.926180 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:01.076179 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:01.076149 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n"] Jul 28 00:53:01.079266 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:53:01.079239 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2034427d_6a25_41c3_9aeb_5e1d4c50f5ec.slice/crio-9ae6e5589b5b97247c1d006a9a7b8ff1218f47ea744533f172c585219a33717f WatchSource:0}: Error finding container 9ae6e5589b5b97247c1d006a9a7b8ff1218f47ea744533f172c585219a33717f: Status 404 returned error can't find the container with id 9ae6e5589b5b97247c1d006a9a7b8ff1218f47ea744533f172c585219a33717f Jul 28 00:53:01.550558 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:01.550518 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" event={"ID":"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec","Type":"ContainerStarted","Data":"9ae6e5589b5b97247c1d006a9a7b8ff1218f47ea744533f172c585219a33717f"} Jul 28 00:53:02.512925 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:02.512900 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-q2wxd" Jul 28 00:53:04.174843 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:04.174804 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 00:53:04.175152 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:04.174876 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 00:53:04.175152 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:04.174902 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 00:53:04.570342 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:04.570311 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" event={"ID":"2034427d-6a25-41c3-9aeb-5e1d4c50f5ec","Type":"ContainerStarted","Data":"d4dc86f2ead3917bde3869e0252d81563db29d42dfefb96d15857e7907c95647"} Jul 28 00:53:04.590947 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:04.590898 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" podStartSLOduration=1.497462114 podStartE2EDuration="4.590885438s" totalImagesPullingTime="3.093423324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:53:04.588271663 +0000 UTC m=+380.349119676" watchObservedRunningTime="2026-07-28 00:53:04.590885438 +0000 UTC m=+380.351733467" Jul 28 00:53:04.926593 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:04.926516 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:05.932435 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:05.932402 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:06.580103 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:06.580069 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:06.581079 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:06.581059 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n" Jul 28 00:53:43.473287 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.473252 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lz4xg"] Jul 28 00:53:43.481280 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.481252 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" Jul 28 00:53:43.483428 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.483405 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-46wtj\"" Jul 28 00:53:43.484443 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.484419 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 00:53:43.484558 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.484457 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 00:53:43.486842 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.486818 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lz4xg"] Jul 28 00:53:43.571722 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.571698 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sm2z\" (UniqueName: \"kubernetes.io/projected/235cc6ad-0ee8-40ff-946b-b02c487b6a41-kube-api-access-7sm2z\") pod \"kuadrant-operator-catalog-lz4xg\" (UID: \"235cc6ad-0ee8-40ff-946b-b02c487b6a41\") " pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" Jul 28 00:53:43.672537 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.672513 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sm2z\" (UniqueName: \"kubernetes.io/projected/235cc6ad-0ee8-40ff-946b-b02c487b6a41-kube-api-access-7sm2z\") pod \"kuadrant-operator-catalog-lz4xg\" (UID: \"235cc6ad-0ee8-40ff-946b-b02c487b6a41\") " pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" Jul 28 00:53:43.679687 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.679667 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sm2z\" (UniqueName: \"kubernetes.io/projected/235cc6ad-0ee8-40ff-946b-b02c487b6a41-kube-api-access-7sm2z\") pod \"kuadrant-operator-catalog-lz4xg\" (UID: \"235cc6ad-0ee8-40ff-946b-b02c487b6a41\") " pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" Jul 28 00:53:43.792129 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.792108 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" Jul 28 00:53:43.858945 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.858908 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lz4xg"] Jul 28 00:53:43.919462 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:43.919438 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lz4xg"] Jul 28 00:53:43.922078 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:53:43.922051 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235cc6ad_0ee8_40ff_946b_b02c487b6a41.slice/crio-fdd1579f2a58b2e522b0899a4c4055d4cfbbb27d3188547773a9b98befa930bd WatchSource:0}: Error finding container fdd1579f2a58b2e522b0899a4c4055d4cfbbb27d3188547773a9b98befa930bd: Status 404 returned error can't find the container with id fdd1579f2a58b2e522b0899a4c4055d4cfbbb27d3188547773a9b98befa930bd Jul 28 00:53:44.050364 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.050311 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ph7q6"] Jul 28 00:53:44.054969 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.054953 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" Jul 28 00:53:44.059901 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.059879 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ph7q6"] Jul 28 00:53:44.176537 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.176516 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjqg\" (UniqueName: \"kubernetes.io/projected/fb01dab4-37cc-4f5b-8c7c-e873bb9c84af-kube-api-access-wvjqg\") pod \"kuadrant-operator-catalog-ph7q6\" (UID: \"fb01dab4-37cc-4f5b-8c7c-e873bb9c84af\") " pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" Jul 28 00:53:44.277568 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.277543 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjqg\" (UniqueName: \"kubernetes.io/projected/fb01dab4-37cc-4f5b-8c7c-e873bb9c84af-kube-api-access-wvjqg\") pod \"kuadrant-operator-catalog-ph7q6\" (UID: \"fb01dab4-37cc-4f5b-8c7c-e873bb9c84af\") " pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" Jul 28 00:53:44.284865 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.284846 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjqg\" (UniqueName: \"kubernetes.io/projected/fb01dab4-37cc-4f5b-8c7c-e873bb9c84af-kube-api-access-wvjqg\") pod \"kuadrant-operator-catalog-ph7q6\" (UID: \"fb01dab4-37cc-4f5b-8c7c-e873bb9c84af\") " pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" Jul 28 00:53:44.365936 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.365870 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" Jul 28 00:53:44.502870 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.502760 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-ph7q6"] Jul 28 00:53:44.505020 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:53:44.504969 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb01dab4_37cc_4f5b_8c7c_e873bb9c84af.slice/crio-dabf13cbb9224f3a70b6547e47734f4e654e53822e7927692758ce9b9fb36642 WatchSource:0}: Error finding container dabf13cbb9224f3a70b6547e47734f4e654e53822e7927692758ce9b9fb36642: Status 404 returned error can't find the container with id dabf13cbb9224f3a70b6547e47734f4e654e53822e7927692758ce9b9fb36642 Jul 28 00:53:44.758612 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.758530 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" event={"ID":"fb01dab4-37cc-4f5b-8c7c-e873bb9c84af","Type":"ContainerStarted","Data":"dabf13cbb9224f3a70b6547e47734f4e654e53822e7927692758ce9b9fb36642"} Jul 28 00:53:44.759642 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:44.759614 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" event={"ID":"235cc6ad-0ee8-40ff-946b-b02c487b6a41","Type":"ContainerStarted","Data":"fdd1579f2a58b2e522b0899a4c4055d4cfbbb27d3188547773a9b98befa930bd"} Jul 28 00:53:46.772627 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:46.772584 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" event={"ID":"235cc6ad-0ee8-40ff-946b-b02c487b6a41","Type":"ContainerStarted","Data":"57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be"} Jul 28 00:53:46.773112 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:46.772622 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" podUID="235cc6ad-0ee8-40ff-946b-b02c487b6a41" containerName="registry-server" containerID="cri-o://57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be" gracePeriod=2 Jul 28 00:53:46.774540 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:46.774501 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" event={"ID":"fb01dab4-37cc-4f5b-8c7c-e873bb9c84af","Type":"ContainerStarted","Data":"e86a8deb2a2c94b38dcc8a37145de75d88b7f396f67cc01d131e401e220830ee"} Jul 28 00:53:46.786183 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:46.786121 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" podStartSLOduration=1.6590699089999998 podStartE2EDuration="3.786103621s" totalImagesPullingTime="2.127033712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:53:46.786086594 +0000 UTC m=+422.546934614" watchObservedRunningTime="2026-07-28 00:53:46.786103621 +0000 UTC m=+422.546951634" Jul 28 00:53:46.800523 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:46.800482 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" podStartSLOduration=1.25381777 podStartE2EDuration="2.800468882s" totalImagesPullingTime="1.546651112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:53:46.798776938 +0000 UTC m=+422.559624952" watchObservedRunningTime="2026-07-28 00:53:46.800468882 +0000 UTC m=+422.561316896" Jul 28 00:53:47.012975 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.012947 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" Jul 28 00:53:47.101271 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.101246 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sm2z\" (UniqueName: \"kubernetes.io/projected/235cc6ad-0ee8-40ff-946b-b02c487b6a41-kube-api-access-7sm2z\") pod \"235cc6ad-0ee8-40ff-946b-b02c487b6a41\" (UID: \"235cc6ad-0ee8-40ff-946b-b02c487b6a41\") " Jul 28 00:53:47.103332 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.103309 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/235cc6ad-0ee8-40ff-946b-b02c487b6a41-kube-api-access-7sm2z" (OuterVolumeSpecName: "kube-api-access-7sm2z") pod "235cc6ad-0ee8-40ff-946b-b02c487b6a41" (UID: "235cc6ad-0ee8-40ff-946b-b02c487b6a41"). InnerVolumeSpecName "kube-api-access-7sm2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:53:47.202158 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.202128 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sm2z\" (UniqueName: \"kubernetes.io/projected/235cc6ad-0ee8-40ff-946b-b02c487b6a41-kube-api-access-7sm2z\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:53:47.781908 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.781869 2550 generic.go:356] "Generic (PLEG): container finished" podID="235cc6ad-0ee8-40ff-946b-b02c487b6a41" containerID="57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be" exitCode=0 Jul 28 00:53:47.782408 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.781937 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" Jul 28 00:53:47.782408 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.781961 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" event={"ID":"235cc6ad-0ee8-40ff-946b-b02c487b6a41","Type":"ContainerDied","Data":"57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be"} Jul 28 00:53:47.782408 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.782039 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lz4xg" event={"ID":"235cc6ad-0ee8-40ff-946b-b02c487b6a41","Type":"ContainerDied","Data":"fdd1579f2a58b2e522b0899a4c4055d4cfbbb27d3188547773a9b98befa930bd"} Jul 28 00:53:47.782408 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.782071 2550 scope.go:117] "RemoveContainer" containerID="57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be" Jul 28 00:53:47.792388 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.792369 2550 scope.go:117] "RemoveContainer" containerID="57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be" Jul 28 00:53:47.792701 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:53:47.792678 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be\": container with ID starting with 57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be not found: ID does not exist" containerID="57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be" Jul 28 00:53:47.792754 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.792711 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be"} err="failed to get container status \"57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be\": rpc error: code = NotFound desc = could not find container \"57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be\": container with ID starting with 57a887a1fc042fa399b362a6f3d3dcb001e163aaa29d73f5184655d93eda98be not found: ID does not exist" Jul 28 00:53:47.802837 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.802813 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lz4xg"] Jul 28 00:53:47.805997 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:47.805972 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lz4xg"] Jul 28 00:53:48.785565 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:48.785528 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="235cc6ad-0ee8-40ff-946b-b02c487b6a41" path="/var/lib/kubelet/pods/235cc6ad-0ee8-40ff-946b-b02c487b6a41/volumes" Jul 28 00:53:54.366185 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:54.366149 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" Jul 28 00:53:54.366673 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:54.366206 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" Jul 28 00:53:54.388886 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:54.388861 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" Jul 28 00:53:54.836787 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:54.836761 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-ph7q6" Jul 28 00:53:59.098386 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.098354 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2"] Jul 28 00:53:59.098818 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.098729 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235cc6ad-0ee8-40ff-946b-b02c487b6a41" containerName="registry-server" Jul 28 00:53:59.098818 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.098741 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="235cc6ad-0ee8-40ff-946b-b02c487b6a41" containerName="registry-server" Jul 28 00:53:59.098818 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.098811 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="235cc6ad-0ee8-40ff-946b-b02c487b6a41" containerName="registry-server" Jul 28 00:53:59.103320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.103300 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.105712 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.105693 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qx2mk\"" Jul 28 00:53:59.111087 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.111062 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2"] Jul 28 00:53:59.288892 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.288860 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.289049 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.288948 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.289049 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.288975 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5smc\" (UniqueName: \"kubernetes.io/projected/fc00c633-50f4-4896-879b-e79c77e8ed62-kube-api-access-l5smc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.390283 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.390218 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.390283 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.390255 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5smc\" (UniqueName: \"kubernetes.io/projected/fc00c633-50f4-4896-879b-e79c77e8ed62-kube-api-access-l5smc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.390466 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.390301 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.390660 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.390639 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.390733 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.390668 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.399028 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.398966 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5smc\" (UniqueName: \"kubernetes.io/projected/fc00c633-50f4-4896-879b-e79c77e8ed62-kube-api-access-l5smc\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.413610 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.413587 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:53:59.546537 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.546513 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2"] Jul 28 00:53:59.548377 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:53:59.548345 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc00c633_50f4_4896_879b_e79c77e8ed62.slice/crio-b3187328fd2b85112dce81723ddb4d191d2173906299ec62624d62efaa26506a WatchSource:0}: Error finding container b3187328fd2b85112dce81723ddb4d191d2173906299ec62624d62efaa26506a: Status 404 returned error can't find the container with id b3187328fd2b85112dce81723ddb4d191d2173906299ec62624d62efaa26506a Jul 28 00:53:59.685980 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.685953 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk"] Jul 28 00:53:59.689348 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.689333 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:53:59.696597 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.696569 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk"] Jul 28 00:53:59.794297 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.794268 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlw7d\" (UniqueName: \"kubernetes.io/projected/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-kube-api-access-zlw7d\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:53:59.794412 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.794341 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:53:59.794412 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.794393 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:53:59.839586 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.839561 2550 generic.go:356] "Generic (PLEG): container finished" podID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerID="d14f640829a56a2556616c4a73abcf48766400c2f987cca798dba66613447d33" exitCode=0 Jul 28 00:53:59.839695 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.839605 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" event={"ID":"fc00c633-50f4-4896-879b-e79c77e8ed62","Type":"ContainerDied","Data":"d14f640829a56a2556616c4a73abcf48766400c2f987cca798dba66613447d33"} Jul 28 00:53:59.839695 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.839656 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" event={"ID":"fc00c633-50f4-4896-879b-e79c77e8ed62","Type":"ContainerStarted","Data":"b3187328fd2b85112dce81723ddb4d191d2173906299ec62624d62efaa26506a"} Jul 28 00:53:59.894875 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.894847 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlw7d\" (UniqueName: \"kubernetes.io/projected/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-kube-api-access-zlw7d\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:53:59.894960 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.894903 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:53:59.894960 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.894944 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:53:59.895295 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.895278 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:53:59.895334 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.895302 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:53:59.905671 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:53:59.905626 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlw7d\" (UniqueName: \"kubernetes.io/projected/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-kube-api-access-zlw7d\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:54:00.014078 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.014059 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:54:00.138361 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.138337 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk"] Jul 28 00:54:00.140233 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:54:00.140207 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dd2a2e_4988_4843_a1d9_d9213c6b1a75.slice/crio-ce563212bd86c9bbb5940aacd862bbeb0e0797571ed1db7d1dedf9e4a4eac309 WatchSource:0}: Error finding container ce563212bd86c9bbb5940aacd862bbeb0e0797571ed1db7d1dedf9e4a4eac309: Status 404 returned error can't find the container with id ce563212bd86c9bbb5940aacd862bbeb0e0797571ed1db7d1dedf9e4a4eac309 Jul 28 00:54:00.281951 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.281922 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg"] Jul 28 00:54:00.285447 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.285431 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.292313 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.292291 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg"] Jul 28 00:54:00.398132 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.398106 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.398262 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.398231 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.398326 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.398268 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcqxd\" (UniqueName: \"kubernetes.io/projected/081731a9-98e2-4a0f-848e-d728b6424925-kube-api-access-rcqxd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.498923 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.498864 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.499045 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.498956 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.499045 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.498997 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcqxd\" (UniqueName: \"kubernetes.io/projected/081731a9-98e2-4a0f-848e-d728b6424925-kube-api-access-rcqxd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.499264 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.499247 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.499316 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.499296 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.506775 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.506749 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcqxd\" (UniqueName: \"kubernetes.io/projected/081731a9-98e2-4a0f-848e-d728b6424925-kube-api-access-rcqxd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.601864 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.601833 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:00.683218 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.683194 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr"] Jul 28 00:54:00.686904 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.686884 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.694967 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.694766 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr"] Jul 28 00:54:00.756330 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.756308 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg"] Jul 28 00:54:00.758840 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:54:00.758814 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081731a9_98e2_4a0f_848e_d728b6424925.slice/crio-645b278d973003c15f31a3fd333675894cb5a606da4e2f77f6e0391b617b40c2 WatchSource:0}: Error finding container 645b278d973003c15f31a3fd333675894cb5a606da4e2f77f6e0391b617b40c2: Status 404 returned error can't find the container with id 645b278d973003c15f31a3fd333675894cb5a606da4e2f77f6e0391b617b40c2 Jul 28 00:54:00.801505 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.801475 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.801590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.801518 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drdzl\" (UniqueName: \"kubernetes.io/projected/08c23cc5-be9b-436a-9a9a-04352f7b4714-kube-api-access-drdzl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.801653 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.801636 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.846297 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.846266 2550 generic.go:356] "Generic (PLEG): container finished" podID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerID="6da4a7852d1627f01b93a406845918397d6b25fa21255cc875c09b9391c6a360" exitCode=0 Jul 28 00:54:00.846428 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.846344 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" event={"ID":"fc00c633-50f4-4896-879b-e79c77e8ed62","Type":"ContainerDied","Data":"6da4a7852d1627f01b93a406845918397d6b25fa21255cc875c09b9391c6a360"} Jul 28 00:54:00.848072 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.848048 2550 generic.go:356] "Generic (PLEG): container finished" podID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerID="20828b5a632f383d172f25a76b80ce91947eede50d686f90dfe791487c23e1ac" exitCode=0 Jul 28 00:54:00.848176 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.848144 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" event={"ID":"05dd2a2e-4988-4843-a1d9-d9213c6b1a75","Type":"ContainerDied","Data":"20828b5a632f383d172f25a76b80ce91947eede50d686f90dfe791487c23e1ac"} Jul 28 00:54:00.848176 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.848172 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" event={"ID":"05dd2a2e-4988-4843-a1d9-d9213c6b1a75","Type":"ContainerStarted","Data":"ce563212bd86c9bbb5940aacd862bbeb0e0797571ed1db7d1dedf9e4a4eac309"} Jul 28 00:54:00.849990 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.849969 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" event={"ID":"081731a9-98e2-4a0f-848e-d728b6424925","Type":"ContainerStarted","Data":"afbfb7c9d0752579303f6d68dfb2112de036694d6ea867a9d4acb4bd199d5bfa"} Jul 28 00:54:00.850106 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.850023 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" event={"ID":"081731a9-98e2-4a0f-848e-d728b6424925","Type":"ContainerStarted","Data":"645b278d973003c15f31a3fd333675894cb5a606da4e2f77f6e0391b617b40c2"} Jul 28 00:54:00.902801 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.902775 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.902894 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.902804 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drdzl\" (UniqueName: \"kubernetes.io/projected/08c23cc5-be9b-436a-9a9a-04352f7b4714-kube-api-access-drdzl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.903077 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.903056 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.903187 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.903170 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.903320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.903307 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.910711 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.910692 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drdzl\" (UniqueName: \"kubernetes.io/projected/08c23cc5-be9b-436a-9a9a-04352f7b4714-kube-api-access-drdzl\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:00.997567 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:00.997534 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:01.121208 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.121185 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr"] Jul 28 00:54:01.122838 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:54:01.122811 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c23cc5_be9b_436a_9a9a_04352f7b4714.slice/crio-9bec3c9899255eda2e17c9c6016569447074b25fb0b311bcb4250d3e9f5af016 WatchSource:0}: Error finding container 9bec3c9899255eda2e17c9c6016569447074b25fb0b311bcb4250d3e9f5af016: Status 404 returned error can't find the container with id 9bec3c9899255eda2e17c9c6016569447074b25fb0b311bcb4250d3e9f5af016 Jul 28 00:54:01.857379 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.857342 2550 generic.go:356] "Generic (PLEG): container finished" podID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerID="1b21e8e0a5506a4b943565831919b45aedaf16972195ad44cf978d5b3d589ec7" exitCode=0 Jul 28 00:54:01.857777 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.857445 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" event={"ID":"05dd2a2e-4988-4843-a1d9-d9213c6b1a75","Type":"ContainerDied","Data":"1b21e8e0a5506a4b943565831919b45aedaf16972195ad44cf978d5b3d589ec7"} Jul 28 00:54:01.859395 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.859371 2550 generic.go:356] "Generic (PLEG): container finished" podID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerID="9da06f561feb6e042e1ce198819b7229ec499f5945f6d056b08c23d7f9cac49f" exitCode=0 Jul 28 00:54:01.859508 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.859462 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" event={"ID":"08c23cc5-be9b-436a-9a9a-04352f7b4714","Type":"ContainerDied","Data":"9da06f561feb6e042e1ce198819b7229ec499f5945f6d056b08c23d7f9cac49f"} Jul 28 00:54:01.859565 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.859516 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" event={"ID":"08c23cc5-be9b-436a-9a9a-04352f7b4714","Type":"ContainerStarted","Data":"9bec3c9899255eda2e17c9c6016569447074b25fb0b311bcb4250d3e9f5af016"} Jul 28 00:54:01.861503 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.861477 2550 generic.go:356] "Generic (PLEG): container finished" podID="081731a9-98e2-4a0f-848e-d728b6424925" containerID="afbfb7c9d0752579303f6d68dfb2112de036694d6ea867a9d4acb4bd199d5bfa" exitCode=0 Jul 28 00:54:01.861627 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.861560 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" event={"ID":"081731a9-98e2-4a0f-848e-d728b6424925","Type":"ContainerDied","Data":"afbfb7c9d0752579303f6d68dfb2112de036694d6ea867a9d4acb4bd199d5bfa"} Jul 28 00:54:01.864727 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.864698 2550 generic.go:356] "Generic (PLEG): container finished" podID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerID="106aa49297b44bc20314fca9db097289d9dbd759e286134d4b0cfa516e96ab52" exitCode=0 Jul 28 00:54:01.864854 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:01.864733 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" event={"ID":"fc00c633-50f4-4896-879b-e79c77e8ed62","Type":"ContainerDied","Data":"106aa49297b44bc20314fca9db097289d9dbd759e286134d4b0cfa516e96ab52"} Jul 28 00:54:02.872252 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:02.872227 2550 generic.go:356] "Generic (PLEG): container finished" podID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerID="9f791aebb65700bffaa05faa62dd229052183256d8bca3a6e7bc10f92b0d699d" exitCode=0 Jul 28 00:54:02.872569 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:02.872301 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" event={"ID":"05dd2a2e-4988-4843-a1d9-d9213c6b1a75","Type":"ContainerDied","Data":"9f791aebb65700bffaa05faa62dd229052183256d8bca3a6e7bc10f92b0d699d"} Jul 28 00:54:02.874039 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:02.874000 2550 generic.go:356] "Generic (PLEG): container finished" podID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerID="a6a47db17ac0d9dd488d622fdfd555d1ab2dffa9a318d60eb6b0680010b75cc2" exitCode=0 Jul 28 00:54:02.874129 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:02.874040 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" event={"ID":"08c23cc5-be9b-436a-9a9a-04352f7b4714","Type":"ContainerDied","Data":"a6a47db17ac0d9dd488d622fdfd555d1ab2dffa9a318d60eb6b0680010b75cc2"} Jul 28 00:54:02.875811 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:02.875790 2550 generic.go:356] "Generic (PLEG): container finished" podID="081731a9-98e2-4a0f-848e-d728b6424925" containerID="723f5d22a9d88ea5ee86a1777b01ed9eedba5dd99dcc6b15530231b5f89f8fd2" exitCode=0 Jul 28 00:54:02.875892 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:02.875872 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" event={"ID":"081731a9-98e2-4a0f-848e-d728b6424925","Type":"ContainerDied","Data":"723f5d22a9d88ea5ee86a1777b01ed9eedba5dd99dcc6b15530231b5f89f8fd2"} Jul 28 00:54:03.001666 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.001638 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:54:03.019105 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.019083 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-util\") pod \"fc00c633-50f4-4896-879b-e79c77e8ed62\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " Jul 28 00:54:03.019219 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.019134 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-bundle\") pod \"fc00c633-50f4-4896-879b-e79c77e8ed62\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " Jul 28 00:54:03.019286 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.019233 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5smc\" (UniqueName: \"kubernetes.io/projected/fc00c633-50f4-4896-879b-e79c77e8ed62-kube-api-access-l5smc\") pod \"fc00c633-50f4-4896-879b-e79c77e8ed62\" (UID: \"fc00c633-50f4-4896-879b-e79c77e8ed62\") " Jul 28 00:54:03.020242 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.020187 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-bundle" (OuterVolumeSpecName: "bundle") pod "fc00c633-50f4-4896-879b-e79c77e8ed62" (UID: "fc00c633-50f4-4896-879b-e79c77e8ed62"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:03.022990 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.022945 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc00c633-50f4-4896-879b-e79c77e8ed62-kube-api-access-l5smc" (OuterVolumeSpecName: "kube-api-access-l5smc") pod "fc00c633-50f4-4896-879b-e79c77e8ed62" (UID: "fc00c633-50f4-4896-879b-e79c77e8ed62"). InnerVolumeSpecName "kube-api-access-l5smc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:54:03.025933 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.025906 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-util" (OuterVolumeSpecName: "util") pod "fc00c633-50f4-4896-879b-e79c77e8ed62" (UID: "fc00c633-50f4-4896-879b-e79c77e8ed62"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:03.119894 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.119866 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5smc\" (UniqueName: \"kubernetes.io/projected/fc00c633-50f4-4896-879b-e79c77e8ed62-kube-api-access-l5smc\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:03.119894 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.119895 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-util\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:03.120049 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.119905 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fc00c633-50f4-4896-879b-e79c77e8ed62-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:03.883249 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.883220 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" Jul 28 00:54:03.883249 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.883238 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2" event={"ID":"fc00c633-50f4-4896-879b-e79c77e8ed62","Type":"ContainerDied","Data":"b3187328fd2b85112dce81723ddb4d191d2173906299ec62624d62efaa26506a"} Jul 28 00:54:03.883652 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.883278 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3187328fd2b85112dce81723ddb4d191d2173906299ec62624d62efaa26506a" Jul 28 00:54:03.885374 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.885351 2550 generic.go:356] "Generic (PLEG): container finished" podID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerID="2de7be02fef4b54fde3a0d4abf600d84172fd696e4c9d319e12e45f1d8f32772" exitCode=0 Jul 28 00:54:03.885495 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.885384 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" event={"ID":"08c23cc5-be9b-436a-9a9a-04352f7b4714","Type":"ContainerDied","Data":"2de7be02fef4b54fde3a0d4abf600d84172fd696e4c9d319e12e45f1d8f32772"} Jul 28 00:54:03.887535 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.887512 2550 generic.go:356] "Generic (PLEG): container finished" podID="081731a9-98e2-4a0f-848e-d728b6424925" containerID="2b75cde47ec7ce35e9920ae30cdd404c327922dff2085cfeada82b1ce2bc9643" exitCode=0 Jul 28 00:54:03.887623 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:03.887588 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" event={"ID":"081731a9-98e2-4a0f-848e-d728b6424925","Type":"ContainerDied","Data":"2b75cde47ec7ce35e9920ae30cdd404c327922dff2085cfeada82b1ce2bc9643"} Jul 28 00:54:04.014574 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.014523 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:54:04.027781 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.027761 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-bundle\") pod \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " Jul 28 00:54:04.027888 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.027797 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-util\") pod \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " Jul 28 00:54:04.027888 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.027822 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlw7d\" (UniqueName: \"kubernetes.io/projected/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-kube-api-access-zlw7d\") pod \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\" (UID: \"05dd2a2e-4988-4843-a1d9-d9213c6b1a75\") " Jul 28 00:54:04.028218 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.028195 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-bundle" (OuterVolumeSpecName: "bundle") pod "05dd2a2e-4988-4843-a1d9-d9213c6b1a75" (UID: "05dd2a2e-4988-4843-a1d9-d9213c6b1a75"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:04.029805 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.029781 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-kube-api-access-zlw7d" (OuterVolumeSpecName: "kube-api-access-zlw7d") pod "05dd2a2e-4988-4843-a1d9-d9213c6b1a75" (UID: "05dd2a2e-4988-4843-a1d9-d9213c6b1a75"). InnerVolumeSpecName "kube-api-access-zlw7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:54:04.034682 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.034658 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-util" (OuterVolumeSpecName: "util") pod "05dd2a2e-4988-4843-a1d9-d9213c6b1a75" (UID: "05dd2a2e-4988-4843-a1d9-d9213c6b1a75"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:04.128888 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.128862 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-util\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:04.128888 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.128884 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlw7d\" (UniqueName: \"kubernetes.io/projected/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-kube-api-access-zlw7d\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:04.129035 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.128893 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/05dd2a2e-4988-4843-a1d9-d9213c6b1a75-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:04.894348 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.894319 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" Jul 28 00:54:04.894348 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.894330 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk" event={"ID":"05dd2a2e-4988-4843-a1d9-d9213c6b1a75","Type":"ContainerDied","Data":"ce563212bd86c9bbb5940aacd862bbeb0e0797571ed1db7d1dedf9e4a4eac309"} Jul 28 00:54:04.894741 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:04.894376 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce563212bd86c9bbb5940aacd862bbeb0e0797571ed1db7d1dedf9e4a4eac309" Jul 28 00:54:05.044674 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.044654 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:05.046623 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.046581 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:05.136488 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.136455 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-bundle\") pod \"08c23cc5-be9b-436a-9a9a-04352f7b4714\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " Jul 28 00:54:05.136638 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.136522 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drdzl\" (UniqueName: \"kubernetes.io/projected/08c23cc5-be9b-436a-9a9a-04352f7b4714-kube-api-access-drdzl\") pod \"08c23cc5-be9b-436a-9a9a-04352f7b4714\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " Jul 28 00:54:05.136638 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.136547 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcqxd\" (UniqueName: \"kubernetes.io/projected/081731a9-98e2-4a0f-848e-d728b6424925-kube-api-access-rcqxd\") pod \"081731a9-98e2-4a0f-848e-d728b6424925\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " Jul 28 00:54:05.136638 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.136575 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-bundle\") pod \"081731a9-98e2-4a0f-848e-d728b6424925\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " Jul 28 00:54:05.136818 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.136718 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-util\") pod \"081731a9-98e2-4a0f-848e-d728b6424925\" (UID: \"081731a9-98e2-4a0f-848e-d728b6424925\") " Jul 28 00:54:05.136818 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.136781 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-util\") pod \"08c23cc5-be9b-436a-9a9a-04352f7b4714\" (UID: \"08c23cc5-be9b-436a-9a9a-04352f7b4714\") " Jul 28 00:54:05.137352 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.137323 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-bundle" (OuterVolumeSpecName: "bundle") pod "081731a9-98e2-4a0f-848e-d728b6424925" (UID: "081731a9-98e2-4a0f-848e-d728b6424925"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:05.137352 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.137345 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-bundle" (OuterVolumeSpecName: "bundle") pod "08c23cc5-be9b-436a-9a9a-04352f7b4714" (UID: "08c23cc5-be9b-436a-9a9a-04352f7b4714"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:05.138702 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.138680 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c23cc5-be9b-436a-9a9a-04352f7b4714-kube-api-access-drdzl" (OuterVolumeSpecName: "kube-api-access-drdzl") pod "08c23cc5-be9b-436a-9a9a-04352f7b4714" (UID: "08c23cc5-be9b-436a-9a9a-04352f7b4714"). InnerVolumeSpecName "kube-api-access-drdzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:54:05.138814 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.138741 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/081731a9-98e2-4a0f-848e-d728b6424925-kube-api-access-rcqxd" (OuterVolumeSpecName: "kube-api-access-rcqxd") pod "081731a9-98e2-4a0f-848e-d728b6424925" (UID: "081731a9-98e2-4a0f-848e-d728b6424925"). InnerVolumeSpecName "kube-api-access-rcqxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:54:05.144835 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.144810 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-util" (OuterVolumeSpecName: "util") pod "081731a9-98e2-4a0f-848e-d728b6424925" (UID: "081731a9-98e2-4a0f-848e-d728b6424925"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:05.145726 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.145704 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-util" (OuterVolumeSpecName: "util") pod "08c23cc5-be9b-436a-9a9a-04352f7b4714" (UID: "08c23cc5-be9b-436a-9a9a-04352f7b4714"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:05.237451 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.237427 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcqxd\" (UniqueName: \"kubernetes.io/projected/081731a9-98e2-4a0f-848e-d728b6424925-kube-api-access-rcqxd\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:05.237451 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.237450 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:05.237565 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.237459 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/081731a9-98e2-4a0f-848e-d728b6424925-util\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:05.237565 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.237467 2550 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-util\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:05.237565 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.237474 2550 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/08c23cc5-be9b-436a-9a9a-04352f7b4714-bundle\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:05.237565 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.237481 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drdzl\" (UniqueName: \"kubernetes.io/projected/08c23cc5-be9b-436a-9a9a-04352f7b4714-kube-api-access-drdzl\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:05.902216 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.902187 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" Jul 28 00:54:05.902595 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.902214 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr" event={"ID":"08c23cc5-be9b-436a-9a9a-04352f7b4714","Type":"ContainerDied","Data":"9bec3c9899255eda2e17c9c6016569447074b25fb0b311bcb4250d3e9f5af016"} Jul 28 00:54:05.902595 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.902274 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bec3c9899255eda2e17c9c6016569447074b25fb0b311bcb4250d3e9f5af016" Jul 28 00:54:05.904385 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.904357 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" event={"ID":"081731a9-98e2-4a0f-848e-d728b6424925","Type":"ContainerDied","Data":"645b278d973003c15f31a3fd333675894cb5a606da4e2f77f6e0391b617b40c2"} Jul 28 00:54:05.904515 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.904392 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg" Jul 28 00:54:05.904515 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:05.904398 2550 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="645b278d973003c15f31a3fd333675894cb5a606da4e2f77f6e0391b617b40c2" Jul 28 00:54:15.939528 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939440 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx"] Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939779 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081731a9-98e2-4a0f-848e-d728b6424925" containerName="pull" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939791 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="081731a9-98e2-4a0f-848e-d728b6424925" containerName="pull" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939805 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerName="pull" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939811 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerName="pull" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939821 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerName="pull" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939827 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerName="pull" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939836 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerName="extract" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939841 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerName="extract" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939850 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerName="util" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939856 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerName="util" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939862 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerName="extract" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939867 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerName="extract" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939872 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerName="pull" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939877 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerName="pull" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939884 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerName="util" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939889 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerName="util" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939897 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081731a9-98e2-4a0f-848e-d728b6424925" containerName="extract" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939902 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="081731a9-98e2-4a0f-848e-d728b6424925" containerName="extract" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939910 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="081731a9-98e2-4a0f-848e-d728b6424925" containerName="util" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939915 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="081731a9-98e2-4a0f-848e-d728b6424925" containerName="util" Jul 28 00:54:15.939909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939921 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerName="extract" Jul 28 00:54:15.940548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939927 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerName="extract" Jul 28 00:54:15.940548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939934 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerName="util" Jul 28 00:54:15.940548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939940 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerName="util" Jul 28 00:54:15.940548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939991 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="081731a9-98e2-4a0f-848e-d728b6424925" containerName="extract" Jul 28 00:54:15.940548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.939999 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="05dd2a2e-4988-4843-a1d9-d9213c6b1a75" containerName="extract" Jul 28 00:54:15.940548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.940020 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="08c23cc5-be9b-436a-9a9a-04352f7b4714" containerName="extract" Jul 28 00:54:15.940548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.940028 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc00c633-50f4-4896-879b-e79c77e8ed62" containerName="extract" Jul 28 00:54:15.943084 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.943066 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" Jul 28 00:54:15.947862 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.947840 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mwkzh\"" Jul 28 00:54:15.968397 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:15.968367 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx"] Jul 28 00:54:16.015584 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:16.015551 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfhbm\" (UniqueName: \"kubernetes.io/projected/70f6a41b-664a-489e-9a9e-4236e881af27-kube-api-access-qfhbm\") pod \"authorino-operator-75cc8d5cb7-qhvjx\" (UID: \"70f6a41b-664a-489e-9a9e-4236e881af27\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" Jul 28 00:54:16.116139 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:16.116114 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfhbm\" (UniqueName: \"kubernetes.io/projected/70f6a41b-664a-489e-9a9e-4236e881af27-kube-api-access-qfhbm\") pod \"authorino-operator-75cc8d5cb7-qhvjx\" (UID: \"70f6a41b-664a-489e-9a9e-4236e881af27\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" Jul 28 00:54:16.140568 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:16.140549 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfhbm\" (UniqueName: \"kubernetes.io/projected/70f6a41b-664a-489e-9a9e-4236e881af27-kube-api-access-qfhbm\") pod \"authorino-operator-75cc8d5cb7-qhvjx\" (UID: \"70f6a41b-664a-489e-9a9e-4236e881af27\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" Jul 28 00:54:16.253655 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:16.253588 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" Jul 28 00:54:16.383782 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:16.383753 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx"] Jul 28 00:54:16.385547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:54:16.385520 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f6a41b_664a_489e_9a9e_4236e881af27.slice/crio-4161c70cdf4731ccbe6ff67bd5998a903afeca1ff595f97bfd0f030b0e69b49f WatchSource:0}: Error finding container 4161c70cdf4731ccbe6ff67bd5998a903afeca1ff595f97bfd0f030b0e69b49f: Status 404 returned error can't find the container with id 4161c70cdf4731ccbe6ff67bd5998a903afeca1ff595f97bfd0f030b0e69b49f Jul 28 00:54:16.968083 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:16.968046 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" event={"ID":"70f6a41b-664a-489e-9a9e-4236e881af27","Type":"ContainerStarted","Data":"4161c70cdf4731ccbe6ff67bd5998a903afeca1ff595f97bfd0f030b0e69b49f"} Jul 28 00:54:18.979610 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:18.979526 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" event={"ID":"70f6a41b-664a-489e-9a9e-4236e881af27","Type":"ContainerStarted","Data":"4d77902d48814a0d616df0e15c5190cde62bd1d235b4ea41a950ffae7e75b34e"} Jul 28 00:54:18.979952 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:18.979640 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" Jul 28 00:54:19.005088 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:19.005043 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" podStartSLOduration=1.811633684 podStartE2EDuration="4.005030011s" totalImagesPullingTime="2.193396327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:19.003566511 +0000 UTC m=+454.764414522" watchObservedRunningTime="2026-07-28 00:54:19.005030011 +0000 UTC m=+454.765878023" Jul 28 00:54:21.708117 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.708084 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm"] Jul 28 00:54:21.714670 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.714650 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:21.717855 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.717831 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b5q2g\"" Jul 28 00:54:21.732489 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.732463 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm"] Jul 28 00:54:21.757326 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.757301 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7s4\" (UniqueName: \"kubernetes.io/projected/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-kube-api-access-rj7s4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" (UID: \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:21.757423 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.757338 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" (UID: \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:21.858175 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.858149 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" (UID: \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:21.858267 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.858216 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7s4\" (UniqueName: \"kubernetes.io/projected/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-kube-api-access-rj7s4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" (UID: \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:21.858489 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.858471 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" (UID: \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:21.867390 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:21.867366 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7s4\" (UniqueName: \"kubernetes.io/projected/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-kube-api-access-rj7s4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" (UID: \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:22.025380 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:22.025321 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:22.173624 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:22.173601 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm"] Jul 28 00:54:22.175831 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:54:22.175802 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451cdaab_3f6d_4d8f_a1d7_bf3420c085b0.slice/crio-876c06bb2381b82d5b9772cfb37c7ba2e39c62d4ebf096dc8772c387d290f7b4 WatchSource:0}: Error finding container 876c06bb2381b82d5b9772cfb37c7ba2e39c62d4ebf096dc8772c387d290f7b4: Status 404 returned error can't find the container with id 876c06bb2381b82d5b9772cfb37c7ba2e39c62d4ebf096dc8772c387d290f7b4 Jul 28 00:54:23.000815 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:23.000780 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" event={"ID":"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0","Type":"ContainerStarted","Data":"876c06bb2381b82d5b9772cfb37c7ba2e39c62d4ebf096dc8772c387d290f7b4"} Jul 28 00:54:23.723934 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:23.723898 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v"] Jul 28 00:54:23.727241 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:23.727220 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" Jul 28 00:54:23.729872 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:23.729851 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-smpt2\"" Jul 28 00:54:23.730237 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:23.730217 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 28 00:54:23.740275 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:23.740255 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v"] Jul 28 00:54:23.775232 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:23.775207 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhdwv\" (UniqueName: \"kubernetes.io/projected/c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85-kube-api-access-mhdwv\") pod \"dns-operator-controller-manager-864dd78c4c-zdh6v\" (UID: \"c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" Jul 28 00:54:23.875897 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:23.875870 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhdwv\" (UniqueName: \"kubernetes.io/projected/c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85-kube-api-access-mhdwv\") pod \"dns-operator-controller-manager-864dd78c4c-zdh6v\" (UID: \"c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" Jul 28 00:54:23.891466 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:23.891440 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhdwv\" (UniqueName: \"kubernetes.io/projected/c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85-kube-api-access-mhdwv\") pod \"dns-operator-controller-manager-864dd78c4c-zdh6v\" (UID: \"c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" Jul 28 00:54:24.039461 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:24.039443 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" Jul 28 00:54:24.206912 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:24.206889 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v"] Jul 28 00:54:24.209466 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:54:24.209437 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cc8e8d_b557_4b87_a9df_d6c9b8cfaa85.slice/crio-0773355c8c40a7084caac0b220a5be4b8705cef9d0365a901979dccf754b4289 WatchSource:0}: Error finding container 0773355c8c40a7084caac0b220a5be4b8705cef9d0365a901979dccf754b4289: Status 404 returned error can't find the container with id 0773355c8c40a7084caac0b220a5be4b8705cef9d0365a901979dccf754b4289 Jul 28 00:54:25.012325 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:25.012288 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" event={"ID":"c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85","Type":"ContainerStarted","Data":"0773355c8c40a7084caac0b220a5be4b8705cef9d0365a901979dccf754b4289"} Jul 28 00:54:27.027906 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:27.027873 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" event={"ID":"c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85","Type":"ContainerStarted","Data":"2b87576ff1ec92c22735cfb35ec37293df616f94c5d55135edae9b1a645970ab"} Jul 28 00:54:27.028383 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:27.027991 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" Jul 28 00:54:27.070983 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:27.070930 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" podStartSLOduration=1.794798397 podStartE2EDuration="4.070917633s" totalImagesPullingTime="2.276119236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:27.070450766 +0000 UTC m=+462.831298781" watchObservedRunningTime="2026-07-28 00:54:27.070917633 +0000 UTC m=+462.831765645" Jul 28 00:54:29.039667 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:29.039629 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" event={"ID":"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0","Type":"ContainerStarted","Data":"c8cfa461ba3ce30423cd1a335f81dbcab32ad8bca17cbb060188052f642fb37f"} Jul 28 00:54:29.040071 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:29.039771 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:29.062464 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:29.062409 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" podStartSLOduration=1.920124106 podStartE2EDuration="8.06239158s" totalImagesPullingTime="6.142267474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:29.061311678 +0000 UTC m=+464.822159691" watchObservedRunningTime="2026-07-28 00:54:29.06239158 +0000 UTC m=+464.823239594" Jul 28 00:54:29.987147 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:29.987116 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qhvjx" Jul 28 00:54:31.156354 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.156323 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn"] Jul 28 00:54:31.161620 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.161591 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.164583 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.164557 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 28 00:54:31.164735 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.164712 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qx2mk\"" Jul 28 00:54:31.164853 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.164834 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 28 00:54:31.175451 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.175426 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn"] Jul 28 00:54:31.234569 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.234538 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt7sd\" (UniqueName: \"kubernetes.io/projected/c0787303-0655-4bf8-87a9-92c6c7ad398c-kube-api-access-bt7sd\") pod \"kuadrant-console-plugin-66b5774985-bb2jn\" (UID: \"c0787303-0655-4bf8-87a9-92c6c7ad398c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.234689 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.234595 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0787303-0655-4bf8-87a9-92c6c7ad398c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bb2jn\" (UID: \"c0787303-0655-4bf8-87a9-92c6c7ad398c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.234689 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.234625 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c0787303-0655-4bf8-87a9-92c6c7ad398c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bb2jn\" (UID: \"c0787303-0655-4bf8-87a9-92c6c7ad398c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.335646 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.335616 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0787303-0655-4bf8-87a9-92c6c7ad398c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bb2jn\" (UID: \"c0787303-0655-4bf8-87a9-92c6c7ad398c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.335646 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.335646 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c0787303-0655-4bf8-87a9-92c6c7ad398c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bb2jn\" (UID: \"c0787303-0655-4bf8-87a9-92c6c7ad398c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.335796 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.335692 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt7sd\" (UniqueName: \"kubernetes.io/projected/c0787303-0655-4bf8-87a9-92c6c7ad398c-kube-api-access-bt7sd\") pod \"kuadrant-console-plugin-66b5774985-bb2jn\" (UID: \"c0787303-0655-4bf8-87a9-92c6c7ad398c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.336329 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.336312 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c0787303-0655-4bf8-87a9-92c6c7ad398c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bb2jn\" (UID: \"c0787303-0655-4bf8-87a9-92c6c7ad398c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.337947 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.337923 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0787303-0655-4bf8-87a9-92c6c7ad398c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bb2jn\" (UID: \"c0787303-0655-4bf8-87a9-92c6c7ad398c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.342971 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.342952 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt7sd\" (UniqueName: \"kubernetes.io/projected/c0787303-0655-4bf8-87a9-92c6c7ad398c-kube-api-access-bt7sd\") pod \"kuadrant-console-plugin-66b5774985-bb2jn\" (UID: \"c0787303-0655-4bf8-87a9-92c6c7ad398c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.470984 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.470932 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" Jul 28 00:54:31.625681 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:31.625657 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn"] Jul 28 00:54:31.627547 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:54:31.627510 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0787303_0655_4bf8_87a9_92c6c7ad398c.slice/crio-446e80a0f393b1afacef8ed773e6e8e1f523720f4e499480a45fbdf186de05f0 WatchSource:0}: Error finding container 446e80a0f393b1afacef8ed773e6e8e1f523720f4e499480a45fbdf186de05f0: Status 404 returned error can't find the container with id 446e80a0f393b1afacef8ed773e6e8e1f523720f4e499480a45fbdf186de05f0 Jul 28 00:54:32.057767 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:32.057735 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" event={"ID":"c0787303-0655-4bf8-87a9-92c6c7ad398c","Type":"ContainerStarted","Data":"446e80a0f393b1afacef8ed773e6e8e1f523720f4e499480a45fbdf186de05f0"} Jul 28 00:54:38.036078 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:38.036048 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-zdh6v" Jul 28 00:54:40.048742 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:40.048713 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:41.900272 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:41.900233 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm"] Jul 28 00:54:41.900723 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:41.900457 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" podUID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" containerName="manager" containerID="cri-o://c8cfa461ba3ce30423cd1a335f81dbcab32ad8bca17cbb060188052f642fb37f" gracePeriod=2 Jul 28 00:54:41.914545 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:41.914520 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm"] Jul 28 00:54:41.967732 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:41.967707 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh"] Jul 28 00:54:41.968229 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:41.968207 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" containerName="manager" Jul 28 00:54:41.968229 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:41.968228 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" containerName="manager" Jul 28 00:54:41.968399 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:41.968320 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" containerName="manager" Jul 28 00:54:41.979426 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:41.979400 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:54:41.994058 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:41.993982 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh"] Jul 28 00:54:42.020725 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:54:42.020687 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" is forbidden: User \"system:node:ip-10-0-137-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-173.ec2.internal' and this object" podUID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:42.133825 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:42.133798 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hvtg\" (UniqueName: \"kubernetes.io/projected/53904bd7-96c2-4d81-9209-33ffc11d1ec7-kube-api-access-4hvtg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q2fqh\" (UID: \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:54:42.133988 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:42.133843 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53904bd7-96c2-4d81-9209-33ffc11d1ec7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q2fqh\" (UID: \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:54:42.234681 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:42.234600 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hvtg\" (UniqueName: \"kubernetes.io/projected/53904bd7-96c2-4d81-9209-33ffc11d1ec7-kube-api-access-4hvtg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q2fqh\" (UID: \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:54:42.234681 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:42.234641 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53904bd7-96c2-4d81-9209-33ffc11d1ec7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q2fqh\" (UID: \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:54:42.234962 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:42.234942 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53904bd7-96c2-4d81-9209-33ffc11d1ec7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q2fqh\" (UID: \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:54:42.258399 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:42.258374 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hvtg\" (UniqueName: \"kubernetes.io/projected/53904bd7-96c2-4d81-9209-33ffc11d1ec7-kube-api-access-4hvtg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q2fqh\" (UID: \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:54:42.317409 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:42.317382 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:54:52.186283 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.186246 2550 generic.go:356] "Generic (PLEG): container finished" podID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" containerID="c8cfa461ba3ce30423cd1a335f81dbcab32ad8bca17cbb060188052f642fb37f" exitCode=0 Jul 28 00:54:52.258074 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.258051 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:52.259944 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:54:52.259914 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" is forbidden: User \"system:node:ip-10-0-137-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-173.ec2.internal' and this object" podUID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:52.273035 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.272997 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh"] Jul 28 00:54:52.273573 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:54:52.273546 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53904bd7_96c2_4d81_9209_33ffc11d1ec7.slice/crio-64cc8fd8e2487ce33b064227c079d281f1c33c59822be0eaed0c6bb828cbf75d WatchSource:0}: Error finding container 64cc8fd8e2487ce33b064227c079d281f1c33c59822be0eaed0c6bb828cbf75d: Status 404 returned error can't find the container with id 64cc8fd8e2487ce33b064227c079d281f1c33c59822be0eaed0c6bb828cbf75d Jul 28 00:54:52.418907 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.418855 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj7s4\" (UniqueName: \"kubernetes.io/projected/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-kube-api-access-rj7s4\") pod \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\" (UID: \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\") " Jul 28 00:54:52.419121 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.418906 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-extensions-socket-volume\") pod \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\" (UID: \"451cdaab-3f6d-4d8f-a1d7-bf3420c085b0\") " Jul 28 00:54:52.419443 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.419420 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" (UID: "451cdaab-3f6d-4d8f-a1d7-bf3420c085b0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:54:52.420712 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.420693 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-kube-api-access-rj7s4" (OuterVolumeSpecName: "kube-api-access-rj7s4") pod "451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" (UID: "451cdaab-3f6d-4d8f-a1d7-bf3420c085b0"). InnerVolumeSpecName "kube-api-access-rj7s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:54:52.519946 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.519920 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-extensions-socket-volume\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:52.519946 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.519945 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7s4\" (UniqueName: \"kubernetes.io/projected/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0-kube-api-access-rj7s4\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:54:52.788794 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:52.788761 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" path="/var/lib/kubelet/pods/451cdaab-3f6d-4d8f-a1d7-bf3420c085b0/volumes" Jul 28 00:54:53.192892 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:53.192854 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" event={"ID":"53904bd7-96c2-4d81-9209-33ffc11d1ec7","Type":"ContainerStarted","Data":"9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764"} Jul 28 00:54:53.193321 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:53.192918 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" event={"ID":"53904bd7-96c2-4d81-9209-33ffc11d1ec7","Type":"ContainerStarted","Data":"64cc8fd8e2487ce33b064227c079d281f1c33c59822be0eaed0c6bb828cbf75d"} Jul 28 00:54:53.193321 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:53.192951 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:54:53.194531 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:53.194507 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" event={"ID":"c0787303-0655-4bf8-87a9-92c6c7ad398c","Type":"ContainerStarted","Data":"b895a6c1518ae329f873195e203f7ba3226c39056318f5b202831f9d8ca98f6b"} Jul 28 00:54:53.195800 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:53.195779 2550 scope.go:117] "RemoveContainer" containerID="c8cfa461ba3ce30423cd1a335f81dbcab32ad8bca17cbb060188052f642fb37f" Jul 28 00:54:53.195909 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:53.195810 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:53.215492 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:53.215449 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" podStartSLOduration=12.215435429 podStartE2EDuration="12.215435429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:53.213667975 +0000 UTC m=+488.974515988" watchObservedRunningTime="2026-07-28 00:54:53.215435429 +0000 UTC m=+488.976283442" Jul 28 00:54:53.215742 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:54:53.215717 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" is forbidden: User \"system:node:ip-10-0-137-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-173.ec2.internal' and this object" podUID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:53.238767 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:53.238722 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bb2jn" podStartSLOduration=1.672990722 podStartE2EDuration="22.238709497s" totalImagesPullingTime="20.565718775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:53.23636166 +0000 UTC m=+488.997209674" watchObservedRunningTime="2026-07-28 00:54:53.238709497 +0000 UTC m=+488.999557510" Jul 28 00:54:54.783265 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:54:54.783226 2550 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7ggdm\" is forbidden: User \"system:node:ip-10-0-137-173.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-173.ec2.internal' and this object" podUID="451cdaab-3f6d-4d8f-a1d7-bf3420c085b0" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7ggdm" Jul 28 00:54:59.955407 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:59.955373 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh"] Jul 28 00:54:59.955832 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:59.955569 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" podUID="53904bd7-96c2-4d81-9209-33ffc11d1ec7" containerName="manager" containerID="cri-o://9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764" gracePeriod=10 Jul 28 00:54:59.957464 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:54:59.957438 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:55:00.197652 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.197629 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:55:00.236044 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.235956 2550 generic.go:356] "Generic (PLEG): container finished" podID="53904bd7-96c2-4d81-9209-33ffc11d1ec7" containerID="9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764" exitCode=0 Jul 28 00:55:00.236044 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.235996 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" Jul 28 00:55:00.236210 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.236046 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" event={"ID":"53904bd7-96c2-4d81-9209-33ffc11d1ec7","Type":"ContainerDied","Data":"9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764"} Jul 28 00:55:00.236210 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.236101 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh" event={"ID":"53904bd7-96c2-4d81-9209-33ffc11d1ec7","Type":"ContainerDied","Data":"64cc8fd8e2487ce33b064227c079d281f1c33c59822be0eaed0c6bb828cbf75d"} Jul 28 00:55:00.236210 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.236123 2550 scope.go:117] "RemoveContainer" containerID="9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764" Jul 28 00:55:00.244207 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.244190 2550 scope.go:117] "RemoveContainer" containerID="9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764" Jul 28 00:55:00.244458 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:55:00.244441 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764\": container with ID starting with 9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764 not found: ID does not exist" containerID="9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764" Jul 28 00:55:00.244503 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.244467 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764"} err="failed to get container status \"9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764\": rpc error: code = NotFound desc = could not find container \"9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764\": container with ID starting with 9140036c85468589b860e8682f723dbc2e60c130ab140327c94064e5af056764 not found: ID does not exist" Jul 28 00:55:00.380681 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.380650 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hvtg\" (UniqueName: \"kubernetes.io/projected/53904bd7-96c2-4d81-9209-33ffc11d1ec7-kube-api-access-4hvtg\") pod \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\" (UID: \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\") " Jul 28 00:55:00.380806 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.380752 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53904bd7-96c2-4d81-9209-33ffc11d1ec7-extensions-socket-volume\") pod \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\" (UID: \"53904bd7-96c2-4d81-9209-33ffc11d1ec7\") " Jul 28 00:55:00.381122 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.381098 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53904bd7-96c2-4d81-9209-33ffc11d1ec7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "53904bd7-96c2-4d81-9209-33ffc11d1ec7" (UID: "53904bd7-96c2-4d81-9209-33ffc11d1ec7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 00:55:00.382584 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.382563 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53904bd7-96c2-4d81-9209-33ffc11d1ec7-kube-api-access-4hvtg" (OuterVolumeSpecName: "kube-api-access-4hvtg") pod "53904bd7-96c2-4d81-9209-33ffc11d1ec7" (UID: "53904bd7-96c2-4d81-9209-33ffc11d1ec7"). InnerVolumeSpecName "kube-api-access-4hvtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:55:00.481839 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.481816 2550 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/53904bd7-96c2-4d81-9209-33ffc11d1ec7-extensions-socket-volume\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:55:00.481839 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.481836 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hvtg\" (UniqueName: \"kubernetes.io/projected/53904bd7-96c2-4d81-9209-33ffc11d1ec7-kube-api-access-4hvtg\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:55:00.558190 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.558158 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh"] Jul 28 00:55:00.563190 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.563166 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q2fqh"] Jul 28 00:55:00.784103 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:00.784077 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53904bd7-96c2-4d81-9209-33ffc11d1ec7" path="/var/lib/kubelet/pods/53904bd7-96c2-4d81-9209-33ffc11d1ec7/volumes" Jul 28 00:55:16.243344 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.243313 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp"] Jul 28 00:55:16.243778 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.243646 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53904bd7-96c2-4d81-9209-33ffc11d1ec7" containerName="manager" Jul 28 00:55:16.243778 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.243658 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="53904bd7-96c2-4d81-9209-33ffc11d1ec7" containerName="manager" Jul 28 00:55:16.243778 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.243719 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="53904bd7-96c2-4d81-9209-33ffc11d1ec7" containerName="manager" Jul 28 00:55:16.971088 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.971050 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp"] Jul 28 00:55:16.971292 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.971131 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:16.973443 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.973423 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-wrfsc\"" Jul 28 00:55:16.990383 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.990361 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:16.990506 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.990391 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26077a5-940c-4f71-9834-a877a480ba27-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:16.990506 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.990422 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:16.990506 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.990440 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:16.990506 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.990456 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbks7\" (UniqueName: \"kubernetes.io/projected/d26077a5-940c-4f71-9834-a877a480ba27-kube-api-access-hbks7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:16.990730 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.990553 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:16.990730 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.990586 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:16.990730 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.990619 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26077a5-940c-4f71-9834-a877a480ba27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:16.990730 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:16.990706 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26077a5-940c-4f71-9834-a877a480ba27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.090964 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.090943 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091081 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.090972 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26077a5-940c-4f71-9834-a877a480ba27-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091081 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.090995 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091081 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091028 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091081 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091051 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbks7\" (UniqueName: \"kubernetes.io/projected/d26077a5-940c-4f71-9834-a877a480ba27-kube-api-access-hbks7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091329 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091096 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091329 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091118 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091329 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091143 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26077a5-940c-4f71-9834-a877a480ba27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091329 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091191 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26077a5-940c-4f71-9834-a877a480ba27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091500 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091367 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091551 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091500 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091551 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091526 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091631 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091568 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.091966 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.091947 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26077a5-940c-4f71-9834-a877a480ba27-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.093327 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.093309 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26077a5-940c-4f71-9834-a877a480ba27-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.093459 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.093441 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26077a5-940c-4f71-9834-a877a480ba27-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.099453 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.099436 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbks7\" (UniqueName: \"kubernetes.io/projected/d26077a5-940c-4f71-9834-a877a480ba27-kube-api-access-hbks7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.099527 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.099454 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26077a5-940c-4f71-9834-a877a480ba27-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9qvnp\" (UID: \"d26077a5-940c-4f71-9834-a877a480ba27\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.285243 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.285180 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:17.438903 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.438878 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp"] Jul 28 00:55:17.441509 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:55:17.441475 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd26077a5_940c_4f71_9834_a877a480ba27.slice/crio-831dd0c3515238247c04cb7b9a7520ff09dfc42983756386037ee5fd4ca4704b WatchSource:0}: Error finding container 831dd0c3515238247c04cb7b9a7520ff09dfc42983756386037ee5fd4ca4704b: Status 404 returned error can't find the container with id 831dd0c3515238247c04cb7b9a7520ff09dfc42983756386037ee5fd4ca4704b Jul 28 00:55:17.443695 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.443655 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 00:55:17.443788 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.443738 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 00:55:17.443844 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:17.443784 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 00:55:18.333087 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:18.333038 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" event={"ID":"d26077a5-940c-4f71-9834-a877a480ba27","Type":"ContainerStarted","Data":"b49007b1cdd635736815a55cb188e458543ad3e0a11f0842e9e15e08ed4ea52a"} Jul 28 00:55:18.333446 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:18.333139 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" event={"ID":"d26077a5-940c-4f71-9834-a877a480ba27","Type":"ContainerStarted","Data":"831dd0c3515238247c04cb7b9a7520ff09dfc42983756386037ee5fd4ca4704b"} Jul 28 00:55:18.351816 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:18.351739 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" podStartSLOduration=2.351722265 podStartE2EDuration="2.351722265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:18.350712467 +0000 UTC m=+514.111560481" watchObservedRunningTime="2026-07-28 00:55:18.351722265 +0000 UTC m=+514.112570280" Jul 28 00:55:19.286229 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.286193 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:19.291651 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.291623 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:19.338320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.338290 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:19.338972 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.338955 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:55:19.367302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.367261 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7hq"] Jul 28 00:55:19.428528 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.428493 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7hq"] Jul 28 00:55:19.428668 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.428581 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:19.431158 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.430976 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 28 00:55:19.463411 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.463380 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7hq"] Jul 28 00:55:19.509957 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.509930 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/060fae9e-a607-458a-b015-0c3c4a3b48d3-config-file\") pod \"limitador-limitador-64c94fd64f-ph7hq\" (UID: \"060fae9e-a607-458a-b015-0c3c4a3b48d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:19.510101 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.509981 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j96wz\" (UniqueName: \"kubernetes.io/projected/060fae9e-a607-458a-b015-0c3c4a3b48d3-kube-api-access-j96wz\") pod \"limitador-limitador-64c94fd64f-ph7hq\" (UID: \"060fae9e-a607-458a-b015-0c3c4a3b48d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:19.611302 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.611273 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j96wz\" (UniqueName: \"kubernetes.io/projected/060fae9e-a607-458a-b015-0c3c4a3b48d3-kube-api-access-j96wz\") pod \"limitador-limitador-64c94fd64f-ph7hq\" (UID: \"060fae9e-a607-458a-b015-0c3c4a3b48d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:19.611440 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.611332 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/060fae9e-a607-458a-b015-0c3c4a3b48d3-config-file\") pod \"limitador-limitador-64c94fd64f-ph7hq\" (UID: \"060fae9e-a607-458a-b015-0c3c4a3b48d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:19.612254 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.612230 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/060fae9e-a607-458a-b015-0c3c4a3b48d3-config-file\") pod \"limitador-limitador-64c94fd64f-ph7hq\" (UID: \"060fae9e-a607-458a-b015-0c3c4a3b48d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:19.618640 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.618614 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j96wz\" (UniqueName: \"kubernetes.io/projected/060fae9e-a607-458a-b015-0c3c4a3b48d3-kube-api-access-j96wz\") pod \"limitador-limitador-64c94fd64f-ph7hq\" (UID: \"060fae9e-a607-458a-b015-0c3c4a3b48d3\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:19.740594 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.740572 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:19.867680 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:19.867653 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7hq"] Jul 28 00:55:19.869390 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:55:19.869364 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060fae9e_a607_458a_b015_0c3c4a3b48d3.slice/crio-5559bf1c904a42ef770bc278bfc6ca970f9870279b738662d5db7be42d2ff15d WatchSource:0}: Error finding container 5559bf1c904a42ef770bc278bfc6ca970f9870279b738662d5db7be42d2ff15d: Status 404 returned error can't find the container with id 5559bf1c904a42ef770bc278bfc6ca970f9870279b738662d5db7be42d2ff15d Jul 28 00:55:20.345236 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.345209 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-8g7fn"] Jul 28 00:55:20.350054 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.350025 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" event={"ID":"060fae9e-a607-458a-b015-0c3c4a3b48d3","Type":"ContainerStarted","Data":"5559bf1c904a42ef770bc278bfc6ca970f9870279b738662d5db7be42d2ff15d"} Jul 28 00:55:20.350269 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.350250 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8g7fn" Jul 28 00:55:20.352170 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.352150 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8g7fn"] Jul 28 00:55:20.352377 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.352354 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gfzvw\"" Jul 28 00:55:20.419282 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.419252 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8vh\" (UniqueName: \"kubernetes.io/projected/609b669d-8612-4efc-b176-4a24d5bb2789-kube-api-access-2j8vh\") pod \"authorino-75d556bd77-8g7fn\" (UID: \"609b669d-8612-4efc-b176-4a24d5bb2789\") " pod="kuadrant-system/authorino-75d556bd77-8g7fn" Jul 28 00:55:20.520703 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.520675 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8vh\" (UniqueName: \"kubernetes.io/projected/609b669d-8612-4efc-b176-4a24d5bb2789-kube-api-access-2j8vh\") pod \"authorino-75d556bd77-8g7fn\" (UID: \"609b669d-8612-4efc-b176-4a24d5bb2789\") " pod="kuadrant-system/authorino-75d556bd77-8g7fn" Jul 28 00:55:20.527856 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.527834 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8vh\" (UniqueName: \"kubernetes.io/projected/609b669d-8612-4efc-b176-4a24d5bb2789-kube-api-access-2j8vh\") pod \"authorino-75d556bd77-8g7fn\" (UID: \"609b669d-8612-4efc-b176-4a24d5bb2789\") " pod="kuadrant-system/authorino-75d556bd77-8g7fn" Jul 28 00:55:20.660293 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.660210 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8g7fn" Jul 28 00:55:20.848788 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:20.848752 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8g7fn"] Jul 28 00:55:20.852358 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:55:20.851486 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod609b669d_8612_4efc_b176_4a24d5bb2789.slice/crio-7225d59657e60d127a4139c8764bce750375ed9e3b8ec081237adec206b3cd22 WatchSource:0}: Error finding container 7225d59657e60d127a4139c8764bce750375ed9e3b8ec081237adec206b3cd22: Status 404 returned error can't find the container with id 7225d59657e60d127a4139c8764bce750375ed9e3b8ec081237adec206b3cd22 Jul 28 00:55:21.356167 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:21.356134 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8g7fn" event={"ID":"609b669d-8612-4efc-b176-4a24d5bb2789","Type":"ContainerStarted","Data":"7225d59657e60d127a4139c8764bce750375ed9e3b8ec081237adec206b3cd22"} Jul 28 00:55:25.381228 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:25.381184 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8g7fn" event={"ID":"609b669d-8612-4efc-b176-4a24d5bb2789","Type":"ContainerStarted","Data":"d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b"} Jul 28 00:55:25.382774 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:25.382749 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" event={"ID":"060fae9e-a607-458a-b015-0c3c4a3b48d3","Type":"ContainerStarted","Data":"a651bddd9caa48285f97c93e19ca12513165a2d7524cd1469fe50af0906ea71c"} Jul 28 00:55:25.382871 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:25.382832 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:25.394260 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:25.394214 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-8g7fn" podStartSLOduration=1.690484753 podStartE2EDuration="5.394201278s" totalImagesPullingTime="3.703716525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:25.392953917 +0000 UTC m=+521.153801931" watchObservedRunningTime="2026-07-28 00:55:25.394201278 +0000 UTC m=+521.155049291" Jul 28 00:55:25.413127 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:25.413081 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" podStartSLOduration=1.727047311 podStartE2EDuration="6.41307041s" totalImagesPullingTime="4.686023099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:25.411567235 +0000 UTC m=+521.172415250" watchObservedRunningTime="2026-07-28 00:55:25.41307041 +0000 UTC m=+521.173918423" Jul 28 00:55:33.769907 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:33.769868 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7hq"] Jul 28 00:55:33.770328 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:33.770144 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" podUID="060fae9e-a607-458a-b015-0c3c4a3b48d3" containerName="limitador" containerID="cri-o://a651bddd9caa48285f97c93e19ca12513165a2d7524cd1469fe50af0906ea71c" gracePeriod=30 Jul 28 00:55:33.770852 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:33.770830 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:34.434756 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:34.434693 2550 generic.go:356] "Generic (PLEG): container finished" podID="060fae9e-a607-458a-b015-0c3c4a3b48d3" containerID="a651bddd9caa48285f97c93e19ca12513165a2d7524cd1469fe50af0906ea71c" exitCode=0 Jul 28 00:55:34.434881 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:34.434762 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" event={"ID":"060fae9e-a607-458a-b015-0c3c4a3b48d3","Type":"ContainerDied","Data":"a651bddd9caa48285f97c93e19ca12513165a2d7524cd1469fe50af0906ea71c"} Jul 28 00:55:34.726364 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:34.726341 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:34.833564 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:34.833533 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/060fae9e-a607-458a-b015-0c3c4a3b48d3-config-file\") pod \"060fae9e-a607-458a-b015-0c3c4a3b48d3\" (UID: \"060fae9e-a607-458a-b015-0c3c4a3b48d3\") " Jul 28 00:55:34.833910 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:34.833614 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j96wz\" (UniqueName: \"kubernetes.io/projected/060fae9e-a607-458a-b015-0c3c4a3b48d3-kube-api-access-j96wz\") pod \"060fae9e-a607-458a-b015-0c3c4a3b48d3\" (UID: \"060fae9e-a607-458a-b015-0c3c4a3b48d3\") " Jul 28 00:55:34.833910 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:34.833863 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060fae9e-a607-458a-b015-0c3c4a3b48d3-config-file" (OuterVolumeSpecName: "config-file") pod "060fae9e-a607-458a-b015-0c3c4a3b48d3" (UID: "060fae9e-a607-458a-b015-0c3c4a3b48d3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 00:55:34.835643 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:34.835619 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060fae9e-a607-458a-b015-0c3c4a3b48d3-kube-api-access-j96wz" (OuterVolumeSpecName: "kube-api-access-j96wz") pod "060fae9e-a607-458a-b015-0c3c4a3b48d3" (UID: "060fae9e-a607-458a-b015-0c3c4a3b48d3"). InnerVolumeSpecName "kube-api-access-j96wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:55:34.935082 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:34.935059 2550 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/060fae9e-a607-458a-b015-0c3c4a3b48d3-config-file\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:55:34.935082 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:34.935081 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j96wz\" (UniqueName: \"kubernetes.io/projected/060fae9e-a607-458a-b015-0c3c4a3b48d3-kube-api-access-j96wz\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:55:35.441590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.441563 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" event={"ID":"060fae9e-a607-458a-b015-0c3c4a3b48d3","Type":"ContainerDied","Data":"5559bf1c904a42ef770bc278bfc6ca970f9870279b738662d5db7be42d2ff15d"} Jul 28 00:55:35.441590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.441591 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ph7hq" Jul 28 00:55:35.441789 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.441617 2550 scope.go:117] "RemoveContainer" containerID="a651bddd9caa48285f97c93e19ca12513165a2d7524cd1469fe50af0906ea71c" Jul 28 00:55:35.462802 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.462779 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7hq"] Jul 28 00:55:35.464836 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.464817 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ph7hq"] Jul 28 00:55:35.816863 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.816831 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh"] Jul 28 00:55:35.817253 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.817238 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="060fae9e-a607-458a-b015-0c3c4a3b48d3" containerName="limitador" Jul 28 00:55:35.817311 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.817256 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="060fae9e-a607-458a-b015-0c3c4a3b48d3" containerName="limitador" Jul 28 00:55:35.817348 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.817312 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="060fae9e-a607-458a-b015-0c3c4a3b48d3" containerName="limitador" Jul 28 00:55:35.820970 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.820954 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:35.823227 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.823203 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 00:55:35.823342 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.823259 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 28 00:55:35.823342 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.823269 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 00:55:35.823342 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.823320 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-72t7w\"" Jul 28 00:55:35.829073 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.829047 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh"] Jul 28 00:55:35.942619 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.942590 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/203ed36c-b61e-4775-93c0-1d0c3b95b5c2-data\") pod \"postgres-69fb4594cf-zl8xh\" (UID: \"203ed36c-b61e-4775-93c0-1d0c3b95b5c2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:35.942926 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:35.942691 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwrpf\" (UniqueName: \"kubernetes.io/projected/203ed36c-b61e-4775-93c0-1d0c3b95b5c2-kube-api-access-zwrpf\") pod \"postgres-69fb4594cf-zl8xh\" (UID: \"203ed36c-b61e-4775-93c0-1d0c3b95b5c2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:36.043231 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:36.043201 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwrpf\" (UniqueName: \"kubernetes.io/projected/203ed36c-b61e-4775-93c0-1d0c3b95b5c2-kube-api-access-zwrpf\") pod \"postgres-69fb4594cf-zl8xh\" (UID: \"203ed36c-b61e-4775-93c0-1d0c3b95b5c2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:36.043348 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:36.043240 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/203ed36c-b61e-4775-93c0-1d0c3b95b5c2-data\") pod \"postgres-69fb4594cf-zl8xh\" (UID: \"203ed36c-b61e-4775-93c0-1d0c3b95b5c2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:36.043635 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:36.043617 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/203ed36c-b61e-4775-93c0-1d0c3b95b5c2-data\") pod \"postgres-69fb4594cf-zl8xh\" (UID: \"203ed36c-b61e-4775-93c0-1d0c3b95b5c2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:36.057752 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:36.057726 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwrpf\" (UniqueName: \"kubernetes.io/projected/203ed36c-b61e-4775-93c0-1d0c3b95b5c2-kube-api-access-zwrpf\") pod \"postgres-69fb4594cf-zl8xh\" (UID: \"203ed36c-b61e-4775-93c0-1d0c3b95b5c2\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:36.132054 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:36.131985 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:36.257997 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:36.257975 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh"] Jul 28 00:55:36.260215 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:55:36.260191 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203ed36c_b61e_4775_93c0_1d0c3b95b5c2.slice/crio-bb85a3b5e9c2b5d4c29eeef79478f852a8dcf87ec828f6670901045e76780dfa WatchSource:0}: Error finding container bb85a3b5e9c2b5d4c29eeef79478f852a8dcf87ec828f6670901045e76780dfa: Status 404 returned error can't find the container with id bb85a3b5e9c2b5d4c29eeef79478f852a8dcf87ec828f6670901045e76780dfa Jul 28 00:55:36.449401 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:36.449312 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" event={"ID":"203ed36c-b61e-4775-93c0-1d0c3b95b5c2","Type":"ContainerStarted","Data":"bb85a3b5e9c2b5d4c29eeef79478f852a8dcf87ec828f6670901045e76780dfa"} Jul 28 00:55:36.788018 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:36.787977 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="060fae9e-a607-458a-b015-0c3c4a3b48d3" path="/var/lib/kubelet/pods/060fae9e-a607-458a-b015-0c3c4a3b48d3/volumes" Jul 28 00:55:41.480234 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:41.480186 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" event={"ID":"203ed36c-b61e-4775-93c0-1d0c3b95b5c2","Type":"ContainerStarted","Data":"fd109501252aba72747d1356696ce1429010ff035647c2329c1a2c0451da30df"} Jul 28 00:55:41.480597 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:41.480263 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:41.496113 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:41.496067 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" podStartSLOduration=1.9559894230000001 podStartE2EDuration="6.496053391s" totalImagesPullingTime="4.540063968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:41.493673091 +0000 UTC m=+537.254521104" watchObservedRunningTime="2026-07-28 00:55:41.496053391 +0000 UTC m=+537.256901404" Jul 28 00:55:47.513579 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:47.513543 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zl8xh" Jul 28 00:55:50.201020 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.200963 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8g7fn"] Jul 28 00:55:50.201509 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.201190 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-8g7fn" podUID="609b669d-8612-4efc-b176-4a24d5bb2789" containerName="authorino" containerID="cri-o://d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b" gracePeriod=30 Jul 28 00:55:50.451910 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.451856 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8g7fn" Jul 28 00:55:50.528423 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.528389 2550 generic.go:356] "Generic (PLEG): container finished" podID="609b669d-8612-4efc-b176-4a24d5bb2789" containerID="d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b" exitCode=0 Jul 28 00:55:50.528542 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.528429 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-8g7fn" Jul 28 00:55:50.528542 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.528477 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8g7fn" event={"ID":"609b669d-8612-4efc-b176-4a24d5bb2789","Type":"ContainerDied","Data":"d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b"} Jul 28 00:55:50.528542 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.528518 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-8g7fn" event={"ID":"609b669d-8612-4efc-b176-4a24d5bb2789","Type":"ContainerDied","Data":"7225d59657e60d127a4139c8764bce750375ed9e3b8ec081237adec206b3cd22"} Jul 28 00:55:50.528542 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.528535 2550 scope.go:117] "RemoveContainer" containerID="d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b" Jul 28 00:55:50.536846 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.536830 2550 scope.go:117] "RemoveContainer" containerID="d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b" Jul 28 00:55:50.537111 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:55:50.537092 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b\": container with ID starting with d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b not found: ID does not exist" containerID="d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b" Jul 28 00:55:50.537186 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.537118 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b"} err="failed to get container status \"d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b\": rpc error: code = NotFound desc = could not find container \"d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b\": container with ID starting with d397deae2943e10db53ad22e4a0947b9e4ef412b69b02b562916df885a24f27b not found: ID does not exist" Jul 28 00:55:50.564548 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.564526 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2j8vh\" (UniqueName: \"kubernetes.io/projected/609b669d-8612-4efc-b176-4a24d5bb2789-kube-api-access-2j8vh\") pod \"609b669d-8612-4efc-b176-4a24d5bb2789\" (UID: \"609b669d-8612-4efc-b176-4a24d5bb2789\") " Jul 28 00:55:50.566335 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.566305 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609b669d-8612-4efc-b176-4a24d5bb2789-kube-api-access-2j8vh" (OuterVolumeSpecName: "kube-api-access-2j8vh") pod "609b669d-8612-4efc-b176-4a24d5bb2789" (UID: "609b669d-8612-4efc-b176-4a24d5bb2789"). InnerVolumeSpecName "kube-api-access-2j8vh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:55:50.665471 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.665446 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j8vh\" (UniqueName: \"kubernetes.io/projected/609b669d-8612-4efc-b176-4a24d5bb2789-kube-api-access-2j8vh\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:55:50.849970 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.849942 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8g7fn"] Jul 28 00:55:50.858729 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:50.858706 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-8g7fn"] Jul 28 00:55:52.783719 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:52.783686 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="609b669d-8612-4efc-b176-4a24d5bb2789" path="/var/lib/kubelet/pods/609b669d-8612-4efc-b176-4a24d5bb2789/volumes" Jul 28 00:55:54.711987 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.711952 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-f9cd4fddd-lq9zj"] Jul 28 00:55:54.712374 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.712357 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609b669d-8612-4efc-b176-4a24d5bb2789" containerName="authorino" Jul 28 00:55:54.712417 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.712376 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="609b669d-8612-4efc-b176-4a24d5bb2789" containerName="authorino" Jul 28 00:55:54.712466 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.712457 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="609b669d-8612-4efc-b176-4a24d5bb2789" containerName="authorino" Jul 28 00:55:54.860054 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.860021 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f9cd4fddd-lq9zj"] Jul 28 00:55:54.860211 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.860092 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:55:54.862616 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.862581 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 00:55:54.862616 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.862613 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 00:55:54.862818 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.862712 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zlnnc\"" Jul 28 00:55:54.885028 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.884996 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-fj6fs"] Jul 28 00:55:54.908795 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.908775 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-fj6fs"] Jul 28 00:55:54.908905 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:54.908862 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:55:55.012164 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.012105 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-258t6\" (UniqueName: \"kubernetes.io/projected/9f2f7869-a333-41ee-8b22-f986ab0299af-kube-api-access-258t6\") pod \"maas-controller-7f6f5fc955-fj6fs\" (UID: \"9f2f7869-a333-41ee-8b22-f986ab0299af\") " pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:55:55.012164 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.012152 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f2f7869-a333-41ee-8b22-f986ab0299af-webhook-cert\") pod \"maas-controller-7f6f5fc955-fj6fs\" (UID: \"9f2f7869-a333-41ee-8b22-f986ab0299af\") " pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:55:55.012348 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.012185 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd9cdb02-649e-449c-8621-653edbbf457b-webhook-cert\") pod \"maas-controller-f9cd4fddd-lq9zj\" (UID: \"bd9cdb02-649e-449c-8621-653edbbf457b\") " pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:55:55.012348 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.012238 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jx6\" (UniqueName: \"kubernetes.io/projected/bd9cdb02-649e-449c-8621-653edbbf457b-kube-api-access-r5jx6\") pod \"maas-controller-f9cd4fddd-lq9zj\" (UID: \"bd9cdb02-649e-449c-8621-653edbbf457b\") " pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:55:55.113416 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.113392 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f2f7869-a333-41ee-8b22-f986ab0299af-webhook-cert\") pod \"maas-controller-7f6f5fc955-fj6fs\" (UID: \"9f2f7869-a333-41ee-8b22-f986ab0299af\") " pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:55:55.113526 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.113421 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd9cdb02-649e-449c-8621-653edbbf457b-webhook-cert\") pod \"maas-controller-f9cd4fddd-lq9zj\" (UID: \"bd9cdb02-649e-449c-8621-653edbbf457b\") " pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:55:55.113526 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.113454 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jx6\" (UniqueName: \"kubernetes.io/projected/bd9cdb02-649e-449c-8621-653edbbf457b-kube-api-access-r5jx6\") pod \"maas-controller-f9cd4fddd-lq9zj\" (UID: \"bd9cdb02-649e-449c-8621-653edbbf457b\") " pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:55:55.113526 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.113502 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-258t6\" (UniqueName: \"kubernetes.io/projected/9f2f7869-a333-41ee-8b22-f986ab0299af-kube-api-access-258t6\") pod \"maas-controller-7f6f5fc955-fj6fs\" (UID: \"9f2f7869-a333-41ee-8b22-f986ab0299af\") " pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:55:55.115858 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.115835 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd9cdb02-649e-449c-8621-653edbbf457b-webhook-cert\") pod \"maas-controller-f9cd4fddd-lq9zj\" (UID: \"bd9cdb02-649e-449c-8621-653edbbf457b\") " pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:55:55.115939 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.115863 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f2f7869-a333-41ee-8b22-f986ab0299af-webhook-cert\") pod \"maas-controller-7f6f5fc955-fj6fs\" (UID: \"9f2f7869-a333-41ee-8b22-f986ab0299af\") " pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:55:55.120851 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.120831 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jx6\" (UniqueName: \"kubernetes.io/projected/bd9cdb02-649e-449c-8621-653edbbf457b-kube-api-access-r5jx6\") pod \"maas-controller-f9cd4fddd-lq9zj\" (UID: \"bd9cdb02-649e-449c-8621-653edbbf457b\") " pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:55:55.120966 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.120947 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-258t6\" (UniqueName: \"kubernetes.io/projected/9f2f7869-a333-41ee-8b22-f986ab0299af-kube-api-access-258t6\") pod \"maas-controller-7f6f5fc955-fj6fs\" (UID: \"9f2f7869-a333-41ee-8b22-f986ab0299af\") " pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:55:55.170471 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.170450 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:55:55.218323 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.218291 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:55:55.339540 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.339517 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-f9cd4fddd-lq9zj"] Jul 28 00:55:55.341840 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:55:55.341812 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9cdb02_649e_449c_8621_653edbbf457b.slice/crio-afd1d6cdddcef62cd28cedc51c89a88213f5cdd750a554f9f2906a5a13bab865 WatchSource:0}: Error finding container afd1d6cdddcef62cd28cedc51c89a88213f5cdd750a554f9f2906a5a13bab865: Status 404 returned error can't find the container with id afd1d6cdddcef62cd28cedc51c89a88213f5cdd750a554f9f2906a5a13bab865 Jul 28 00:55:55.388638 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.388616 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-fj6fs"] Jul 28 00:55:55.390555 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:55:55.390525 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2f7869_a333_41ee_8b22_f986ab0299af.slice/crio-c1adf837cd538efff4b491270b56d2fa047c8b133e21d90509319caa6f99610e WatchSource:0}: Error finding container c1adf837cd538efff4b491270b56d2fa047c8b133e21d90509319caa6f99610e: Status 404 returned error can't find the container with id c1adf837cd538efff4b491270b56d2fa047c8b133e21d90509319caa6f99610e Jul 28 00:55:55.555221 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.555158 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" event={"ID":"9f2f7869-a333-41ee-8b22-f986ab0299af","Type":"ContainerStarted","Data":"c1adf837cd538efff4b491270b56d2fa047c8b133e21d90509319caa6f99610e"} Jul 28 00:55:55.556326 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:55.556304 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" event={"ID":"bd9cdb02-649e-449c-8621-653edbbf457b","Type":"ContainerStarted","Data":"afd1d6cdddcef62cd28cedc51c89a88213f5cdd750a554f9f2906a5a13bab865"} Jul 28 00:55:58.579585 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:58.579541 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" event={"ID":"9f2f7869-a333-41ee-8b22-f986ab0299af","Type":"ContainerStarted","Data":"3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf"} Jul 28 00:55:58.580070 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:58.579640 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:55:58.581180 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:58.581148 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" event={"ID":"bd9cdb02-649e-449c-8621-653edbbf457b","Type":"ContainerStarted","Data":"2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06"} Jul 28 00:55:58.581243 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:58.581215 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:55:58.595698 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:58.595661 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" podStartSLOduration=1.972251035 podStartE2EDuration="4.595648606s" totalImagesPullingTime="2.623397571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:58.594593335 +0000 UTC m=+554.355441352" watchObservedRunningTime="2026-07-28 00:55:58.595648606 +0000 UTC m=+554.356496619" Jul 28 00:55:58.611559 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:55:58.611520 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" podStartSLOduration=1.930304662 podStartE2EDuration="4.611508315s" totalImagesPullingTime="2.681203653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:58.60918124 +0000 UTC m=+554.370029235" watchObservedRunningTime="2026-07-28 00:55:58.611508315 +0000 UTC m=+554.372356328" Jul 28 00:56:09.589857 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:09.589811 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:56:09.590383 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:09.590030 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:56:09.655750 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:09.655704 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f9cd4fddd-lq9zj"] Jul 28 00:56:09.656037 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:09.655984 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" podUID="bd9cdb02-649e-449c-8621-653edbbf457b" containerName="manager" containerID="cri-o://2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06" gracePeriod=30 Jul 28 00:56:09.896590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:09.896566 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:56:09.941263 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:09.941237 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5jx6\" (UniqueName: \"kubernetes.io/projected/bd9cdb02-649e-449c-8621-653edbbf457b-kube-api-access-r5jx6\") pod \"bd9cdb02-649e-449c-8621-653edbbf457b\" (UID: \"bd9cdb02-649e-449c-8621-653edbbf457b\") " Jul 28 00:56:09.941391 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:09.941278 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd9cdb02-649e-449c-8621-653edbbf457b-webhook-cert\") pod \"bd9cdb02-649e-449c-8621-653edbbf457b\" (UID: \"bd9cdb02-649e-449c-8621-653edbbf457b\") " Jul 28 00:56:09.943449 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:09.943413 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9cdb02-649e-449c-8621-653edbbf457b-kube-api-access-r5jx6" (OuterVolumeSpecName: "kube-api-access-r5jx6") pod "bd9cdb02-649e-449c-8621-653edbbf457b" (UID: "bd9cdb02-649e-449c-8621-653edbbf457b"). InnerVolumeSpecName "kube-api-access-r5jx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:56:09.943567 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:09.943466 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd9cdb02-649e-449c-8621-653edbbf457b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "bd9cdb02-649e-449c-8621-653edbbf457b" (UID: "bd9cdb02-649e-449c-8621-653edbbf457b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:56:10.042727 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.042702 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5jx6\" (UniqueName: \"kubernetes.io/projected/bd9cdb02-649e-449c-8621-653edbbf457b-kube-api-access-r5jx6\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:56:10.042727 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.042724 2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bd9cdb02-649e-449c-8621-653edbbf457b-webhook-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:56:10.654358 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.654326 2550 generic.go:356] "Generic (PLEG): container finished" podID="bd9cdb02-649e-449c-8621-653edbbf457b" containerID="2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06" exitCode=0 Jul 28 00:56:10.654814 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.654379 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" Jul 28 00:56:10.654814 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.654424 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" event={"ID":"bd9cdb02-649e-449c-8621-653edbbf457b","Type":"ContainerDied","Data":"2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06"} Jul 28 00:56:10.654814 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.654509 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-f9cd4fddd-lq9zj" event={"ID":"bd9cdb02-649e-449c-8621-653edbbf457b","Type":"ContainerDied","Data":"afd1d6cdddcef62cd28cedc51c89a88213f5cdd750a554f9f2906a5a13bab865"} Jul 28 00:56:10.654814 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.654544 2550 scope.go:117] "RemoveContainer" containerID="2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06" Jul 28 00:56:10.664778 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.664576 2550 scope.go:117] "RemoveContainer" containerID="2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06" Jul 28 00:56:10.664889 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:56:10.664870 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06\": container with ID starting with 2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06 not found: ID does not exist" containerID="2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06" Jul 28 00:56:10.664934 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.664898 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06"} err="failed to get container status \"2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06\": rpc error: code = NotFound desc = could not find container \"2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06\": container with ID starting with 2a9c4d4d984ee2c6aa2dff037eb1b1cc7de6ffaf7b87ebb1b119e1a5899d0e06 not found: ID does not exist" Jul 28 00:56:10.677287 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.677261 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-f9cd4fddd-lq9zj"] Jul 28 00:56:10.681500 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.681476 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-f9cd4fddd-lq9zj"] Jul 28 00:56:10.785125 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:10.785100 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd9cdb02-649e-449c-8621-653edbbf457b" path="/var/lib/kubelet/pods/bd9cdb02-649e-449c-8621-653edbbf457b/volumes" Jul 28 00:56:31.205027 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.204979 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx"] Jul 28 00:56:31.205561 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.205478 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd9cdb02-649e-449c-8621-653edbbf457b" containerName="manager" Jul 28 00:56:31.205561 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.205497 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd9cdb02-649e-449c-8621-653edbbf457b" containerName="manager" Jul 28 00:56:31.205641 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.205580 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd9cdb02-649e-449c-8621-653edbbf457b" containerName="manager" Jul 28 00:56:31.209930 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.209913 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.212105 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.212084 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 28 00:56:31.223345 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.223318 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx"] Jul 28 00:56:31.311128 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.311099 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qbv\" (UniqueName: \"kubernetes.io/projected/84308a96-5e6b-40d0-a9f2-e8d7141a1a60-kube-api-access-s5qbv\") pod \"maas-api-7b489644c9-5ckzx\" (UID: \"84308a96-5e6b-40d0-a9f2-e8d7141a1a60\") " pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.311250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.311145 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84308a96-5e6b-40d0-a9f2-e8d7141a1a60-openshift-service-ca\") pod \"maas-api-7b489644c9-5ckzx\" (UID: \"84308a96-5e6b-40d0-a9f2-e8d7141a1a60\") " pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.311250 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.311228 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84308a96-5e6b-40d0-a9f2-e8d7141a1a60-maas-api-tls\") pod \"maas-api-7b489644c9-5ckzx\" (UID: \"84308a96-5e6b-40d0-a9f2-e8d7141a1a60\") " pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.411970 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.411943 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qbv\" (UniqueName: \"kubernetes.io/projected/84308a96-5e6b-40d0-a9f2-e8d7141a1a60-kube-api-access-s5qbv\") pod \"maas-api-7b489644c9-5ckzx\" (UID: \"84308a96-5e6b-40d0-a9f2-e8d7141a1a60\") " pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.412111 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.411979 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84308a96-5e6b-40d0-a9f2-e8d7141a1a60-openshift-service-ca\") pod \"maas-api-7b489644c9-5ckzx\" (UID: \"84308a96-5e6b-40d0-a9f2-e8d7141a1a60\") " pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.412111 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.412040 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84308a96-5e6b-40d0-a9f2-e8d7141a1a60-maas-api-tls\") pod \"maas-api-7b489644c9-5ckzx\" (UID: \"84308a96-5e6b-40d0-a9f2-e8d7141a1a60\") " pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.412675 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.412647 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/84308a96-5e6b-40d0-a9f2-e8d7141a1a60-openshift-service-ca\") pod \"maas-api-7b489644c9-5ckzx\" (UID: \"84308a96-5e6b-40d0-a9f2-e8d7141a1a60\") " pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.414454 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.414432 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/84308a96-5e6b-40d0-a9f2-e8d7141a1a60-maas-api-tls\") pod \"maas-api-7b489644c9-5ckzx\" (UID: \"84308a96-5e6b-40d0-a9f2-e8d7141a1a60\") " pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.420171 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.420149 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qbv\" (UniqueName: \"kubernetes.io/projected/84308a96-5e6b-40d0-a9f2-e8d7141a1a60-kube-api-access-s5qbv\") pod \"maas-api-7b489644c9-5ckzx\" (UID: \"84308a96-5e6b-40d0-a9f2-e8d7141a1a60\") " pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.520958 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.520897 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:31.653578 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.653540 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx"] Jul 28 00:56:31.655514 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:56:31.655476 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84308a96_5e6b_40d0_a9f2_e8d7141a1a60.slice/crio-5478dc0dc1b9e831809f476dfcbc3475d43f693f7a4a8d35b5c10987f3b7d830 WatchSource:0}: Error finding container 5478dc0dc1b9e831809f476dfcbc3475d43f693f7a4a8d35b5c10987f3b7d830: Status 404 returned error can't find the container with id 5478dc0dc1b9e831809f476dfcbc3475d43f693f7a4a8d35b5c10987f3b7d830 Jul 28 00:56:31.729590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.729564 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k"] Jul 28 00:56:31.732471 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.732456 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:31.735871 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.735853 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 28 00:56:31.735966 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.735945 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g9rbm\"" Jul 28 00:56:31.750140 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.750120 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k"] Jul 28 00:56:31.780502 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.780446 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" event={"ID":"84308a96-5e6b-40d0-a9f2-e8d7141a1a60","Type":"ContainerStarted","Data":"5478dc0dc1b9e831809f476dfcbc3475d43f693f7a4a8d35b5c10987f3b7d830"} Jul 28 00:56:31.816040 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.815990 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c644166a-c927-4dc1-b046-30b8097b10af-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-twl5k\" (UID: \"c644166a-c927-4dc1-b046-30b8097b10af\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:31.816147 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.816067 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjl47\" (UniqueName: \"kubernetes.io/projected/c644166a-c927-4dc1-b046-30b8097b10af-kube-api-access-pjl47\") pod \"payload-pre-processing-687dbb6cf6-twl5k\" (UID: \"c644166a-c927-4dc1-b046-30b8097b10af\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:31.917215 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.917189 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c644166a-c927-4dc1-b046-30b8097b10af-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-twl5k\" (UID: \"c644166a-c927-4dc1-b046-30b8097b10af\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:31.917328 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.917227 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjl47\" (UniqueName: \"kubernetes.io/projected/c644166a-c927-4dc1-b046-30b8097b10af-kube-api-access-pjl47\") pod \"payload-pre-processing-687dbb6cf6-twl5k\" (UID: \"c644166a-c927-4dc1-b046-30b8097b10af\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:31.917927 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.917910 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c644166a-c927-4dc1-b046-30b8097b10af-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-twl5k\" (UID: \"c644166a-c927-4dc1-b046-30b8097b10af\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:31.924676 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:31.924659 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjl47\" (UniqueName: \"kubernetes.io/projected/c644166a-c927-4dc1-b046-30b8097b10af-kube-api-access-pjl47\") pod \"payload-pre-processing-687dbb6cf6-twl5k\" (UID: \"c644166a-c927-4dc1-b046-30b8097b10af\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:32.042508 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:32.042475 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:32.171707 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:32.171678 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k"] Jul 28 00:56:32.173869 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:56:32.173844 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc644166a_c927_4dc1_b046_30b8097b10af.slice/crio-bc31c934a0c1ebe716c66a872cc21ff25e678463a77e758289986a127ce7a2b7 WatchSource:0}: Error finding container bc31c934a0c1ebe716c66a872cc21ff25e678463a77e758289986a127ce7a2b7: Status 404 returned error can't find the container with id bc31c934a0c1ebe716c66a872cc21ff25e678463a77e758289986a127ce7a2b7 Jul 28 00:56:32.792568 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:32.792485 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" event={"ID":"c644166a-c927-4dc1-b046-30b8097b10af","Type":"ContainerStarted","Data":"bc31c934a0c1ebe716c66a872cc21ff25e678463a77e758289986a127ce7a2b7"} Jul 28 00:56:33.800205 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:33.800163 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" event={"ID":"84308a96-5e6b-40d0-a9f2-e8d7141a1a60","Type":"ContainerStarted","Data":"f85f73f6d5cb5494b621ceeccfde024a6e0d29c3fbdfe432eb6eddc3ff93ed55"} Jul 28 00:56:33.800577 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:33.800310 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:33.815838 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:33.815632 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" podStartSLOduration=0.832865176 podStartE2EDuration="2.815615132s" totalImagesPullingTime="1.982749956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:56:33.814658955 +0000 UTC m=+589.575506970" watchObservedRunningTime="2026-07-28 00:56:33.815615132 +0000 UTC m=+589.576463176" Jul 28 00:56:34.812720 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:34.812681 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" event={"ID":"c644166a-c927-4dc1-b046-30b8097b10af","Type":"ContainerStarted","Data":"4a85824884d51ecb9146970cc2e7da6fdc700e0e915e8439327e4757993a3e4c"} Jul 28 00:56:34.813326 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:34.813296 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:34.829570 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:34.829519 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" podStartSLOduration=1.7452526229999998 podStartE2EDuration="3.829505858s" totalImagesPullingTime="2.084253235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:56:34.827259439 +0000 UTC m=+590.588107452" watchObservedRunningTime="2026-07-28 00:56:34.829505858 +0000 UTC m=+590.590353871" Jul 28 00:56:39.819398 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:39.819373 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b489644c9-5ckzx" Jul 28 00:56:39.844840 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:39.844812 2550 generic.go:356] "Generic (PLEG): container finished" podID="d26077a5-940c-4f71-9834-a877a480ba27" containerID="b49007b1cdd635736815a55cb188e458543ad3e0a11f0842e9e15e08ed4ea52a" exitCode=0 Jul 28 00:56:39.844962 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:39.844842 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" event={"ID":"d26077a5-940c-4f71-9834-a877a480ba27","Type":"ContainerDied","Data":"b49007b1cdd635736815a55cb188e458543ad3e0a11f0842e9e15e08ed4ea52a"} Jul 28 00:56:39.845156 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:39.845141 2550 scope.go:117] "RemoveContainer" containerID="b49007b1cdd635736815a55cb188e458543ad3e0a11f0842e9e15e08ed4ea52a" Jul 28 00:56:39.846054 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:39.846028 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 00:56:39.846114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:39.846089 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 00:56:39.846149 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:39.846123 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 00:56:40.879618 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:40.879575 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" event={"ID":"d26077a5-940c-4f71-9834-a877a480ba27","Type":"ContainerStarted","Data":"a4da59247ddfec3c7c6f5ffa2b4c0316beaa041e36432e48bc0c13418f154913"} Jul 28 00:56:41.285990 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:41.285908 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:56:41.286193 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:41.286171 2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9qvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 28 00:56:41.286255 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:41.286236 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" podUID="d26077a5-940c-4f71-9834-a877a480ba27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 28 00:56:42.285976 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:42.285939 2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9qvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 28 00:56:42.286391 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:42.286000 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" podUID="d26077a5-940c-4f71-9834-a877a480ba27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 28 00:56:43.285785 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:43.285751 2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9qvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 28 00:56:43.285938 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:43.285805 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" podUID="d26077a5-940c-4f71-9834-a877a480ba27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 28 00:56:44.286320 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:44.286281 2550 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9qvnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 28 00:56:44.286791 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:44.286341 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" podUID="d26077a5-940c-4f71-9834-a877a480ba27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 28 00:56:45.290367 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:45.290333 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:56:45.290805 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:45.290629 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:56:45.291317 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:45.291297 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9qvnp" Jul 28 00:56:45.820712 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:45.820673 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-twl5k" Jul 28 00:56:53.723129 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.723093 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b"] Jul 28 00:56:53.727725 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.727703 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.730067 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.730048 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 28 00:56:53.730936 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.730899 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 28 00:56:53.730936 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.730923 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l9t28\"" Jul 28 00:56:53.731182 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.731093 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 28 00:56:53.735384 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.735363 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b"] Jul 28 00:56:53.799920 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.799898 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.800049 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.799952 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11199715-b494-44fe-bb7b-0cb83e69578e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.800049 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.799969 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22sx\" (UniqueName: \"kubernetes.io/projected/11199715-b494-44fe-bb7b-0cb83e69578e-kube-api-access-h22sx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.800134 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.800046 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.800134 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.800088 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.800134 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.800105 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.901323 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.901289 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.901465 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.901337 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.901465 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.901358 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.901465 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.901383 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.901465 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.901437 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11199715-b494-44fe-bb7b-0cb83e69578e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.901465 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.901453 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22sx\" (UniqueName: \"kubernetes.io/projected/11199715-b494-44fe-bb7b-0cb83e69578e-kube-api-access-h22sx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.901728 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.901682 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.901799 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.901746 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.901799 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.901764 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.903637 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.903609 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11199715-b494-44fe-bb7b-0cb83e69578e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.903887 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.903870 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11199715-b494-44fe-bb7b-0cb83e69578e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:53.909475 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:53.909454 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22sx\" (UniqueName: \"kubernetes.io/projected/11199715-b494-44fe-bb7b-0cb83e69578e-kube-api-access-h22sx\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b\" (UID: \"11199715-b494-44fe-bb7b-0cb83e69578e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:54.041634 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:54.041614 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:56:54.177140 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:54.177099 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b"] Jul 28 00:56:54.178889 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:56:54.178850 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11199715_b494_44fe_bb7b_0cb83e69578e.slice/crio-92d277a7f72ba3534aa02bf3c88f5ddef32f651a714a1196e58e9540254c93c8 WatchSource:0}: Error finding container 92d277a7f72ba3534aa02bf3c88f5ddef32f651a714a1196e58e9540254c93c8: Status 404 returned error can't find the container with id 92d277a7f72ba3534aa02bf3c88f5ddef32f651a714a1196e58e9540254c93c8 Jul 28 00:56:54.180439 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:54.180420 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 00:56:54.963472 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:54.963386 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" event={"ID":"11199715-b494-44fe-bb7b-0cb83e69578e","Type":"ContainerStarted","Data":"92d277a7f72ba3534aa02bf3c88f5ddef32f651a714a1196e58e9540254c93c8"} Jul 28 00:56:55.971598 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:55.971550 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" event={"ID":"11199715-b494-44fe-bb7b-0cb83e69578e","Type":"ContainerStarted","Data":"2d2f4c3ae15541622c4c847d29bc5b07640a2528bf5a35a0af64e06569666ec1"} Jul 28 00:56:55.990065 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:56:55.989995 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" podStartSLOduration=1.898199134 podStartE2EDuration="2.989978418s" totalImagesPullingTime="1.091779284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:56:55.986886496 +0000 UTC m=+611.747734505" watchObservedRunningTime="2026-07-28 00:56:55.989978418 +0000 UTC m=+611.750826430" Jul 28 00:57:04.042121 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:04.042079 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:57:04.042671 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:04.042160 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:57:04.054845 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:04.054826 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:57:05.033187 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:05.033153 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b" Jul 28 00:57:16.825290 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.825210 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw"] Jul 28 00:57:16.833577 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.833559 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:16.837627 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.837601 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 28 00:57:16.840228 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.840203 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw"] Jul 28 00:57:16.986340 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.986306 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:16.986586 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.986362 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:16.986586 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.986412 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:16.986586 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.986438 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7slb\" (UniqueName: \"kubernetes.io/projected/b3616050-ec5a-46d5-9c16-731518c7047d-kube-api-access-w7slb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:16.986586 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.986499 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:16.986586 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:16.986571 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3616050-ec5a-46d5-9c16-731518c7047d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.087554 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.087482 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3616050-ec5a-46d5-9c16-731518c7047d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.087554 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.087544 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.087718 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.087588 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.087718 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.087641 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.087718 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.087665 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7slb\" (UniqueName: \"kubernetes.io/projected/b3616050-ec5a-46d5-9c16-731518c7047d-kube-api-access-w7slb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.087718 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.087709 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.088087 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.088066 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.088179 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.088163 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.088361 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.088339 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.089908 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.089887 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3616050-ec5a-46d5-9c16-731518c7047d-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.089993 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.089973 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3616050-ec5a-46d5-9c16-731518c7047d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.095364 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.095346 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7slb\" (UniqueName: \"kubernetes.io/projected/b3616050-ec5a-46d5-9c16-731518c7047d-kube-api-access-w7slb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zp9vw\" (UID: \"b3616050-ec5a-46d5-9c16-731518c7047d\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.144466 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.144444 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:17.277420 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:17.277391 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw"] Jul 28 00:57:17.278133 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:57:17.278107 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3616050_ec5a_46d5_9c16_731518c7047d.slice/crio-563e2b718ca96c911c89003715312b1a5b44f725439e59007448cadd13ebf01f WatchSource:0}: Error finding container 563e2b718ca96c911c89003715312b1a5b44f725439e59007448cadd13ebf01f: Status 404 returned error can't find the container with id 563e2b718ca96c911c89003715312b1a5b44f725439e59007448cadd13ebf01f Jul 28 00:57:18.097989 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:18.097942 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" event={"ID":"b3616050-ec5a-46d5-9c16-731518c7047d","Type":"ContainerStarted","Data":"1421ff88a80c00e42958ba7e5d68673300385f29e3fc5d640b7675d9777a4bb8"} Jul 28 00:57:18.098509 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:18.098036 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" event={"ID":"b3616050-ec5a-46d5-9c16-731518c7047d","Type":"ContainerStarted","Data":"563e2b718ca96c911c89003715312b1a5b44f725439e59007448cadd13ebf01f"} Jul 28 00:57:18.115466 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:18.115423 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" podStartSLOduration=1.951610171 podStartE2EDuration="2.115407654s" totalImagesPullingTime="163.797483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:57:18.114866531 +0000 UTC m=+633.875714547" watchObservedRunningTime="2026-07-28 00:57:18.115407654 +0000 UTC m=+633.876255666" Jul 28 00:57:21.920525 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:21.920480 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk"] Jul 28 00:57:21.924967 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:21.924947 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:21.927114 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:21.927090 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 28 00:57:21.931934 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:21.931907 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk"] Jul 28 00:57:22.025366 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.025337 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.025465 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.025374 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.025465 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.025396 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96fa59c-be24-476e-b476-fb81598366d0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.025579 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.025470 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.025579 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.025529 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.025579 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.025564 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhj4k\" (UniqueName: \"kubernetes.io/projected/d96fa59c-be24-476e-b476-fb81598366d0-kube-api-access-qhj4k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.126437 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.126404 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.126590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.126443 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhj4k\" (UniqueName: \"kubernetes.io/projected/d96fa59c-be24-476e-b476-fb81598366d0-kube-api-access-qhj4k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.126590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.126489 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.126590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.126528 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.126590 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.126571 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96fa59c-be24-476e-b476-fb81598366d0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.126808 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.126659 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.126892 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.126869 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.126951 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.126892 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.127128 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.127098 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.128789 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.128761 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d96fa59c-be24-476e-b476-fb81598366d0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.128881 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.128859 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d96fa59c-be24-476e-b476-fb81598366d0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.133825 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.133806 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhj4k\" (UniqueName: \"kubernetes.io/projected/d96fa59c-be24-476e-b476-fb81598366d0-kube-api-access-qhj4k\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk\" (UID: \"d96fa59c-be24-476e-b476-fb81598366d0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.237660 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.237592 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:22.365581 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:22.365540 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk"] Jul 28 00:57:22.367073 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:57:22.367047 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd96fa59c_be24_476e_b476_fb81598366d0.slice/crio-4c48f0e69a2e2de2c46cb6de7a533f5a2a68246f4e5fbb7d243c154f051d06cd WatchSource:0}: Error finding container 4c48f0e69a2e2de2c46cb6de7a533f5a2a68246f4e5fbb7d243c154f051d06cd: Status 404 returned error can't find the container with id 4c48f0e69a2e2de2c46cb6de7a533f5a2a68246f4e5fbb7d243c154f051d06cd Jul 28 00:57:23.126132 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:23.126091 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" event={"ID":"d96fa59c-be24-476e-b476-fb81598366d0","Type":"ContainerStarted","Data":"359d6257101770dfb7ce65e8c229e793af68a20a393302994c4437dc10146a33"} Jul 28 00:57:23.126505 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:23.126159 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" event={"ID":"d96fa59c-be24-476e-b476-fb81598366d0","Type":"ContainerStarted","Data":"4c48f0e69a2e2de2c46cb6de7a533f5a2a68246f4e5fbb7d243c154f051d06cd"} Jul 28 00:57:23.142783 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:23.142729 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" podStartSLOduration=1.969988894 podStartE2EDuration="2.142713714s" totalImagesPullingTime="172.72482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:57:23.141972296 +0000 UTC m=+638.902820310" watchObservedRunningTime="2026-07-28 00:57:23.142713714 +0000 UTC m=+638.903561726" Jul 28 00:57:27.144637 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:27.144580 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:27.145082 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:27.144693 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:27.157255 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:27.157229 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:28.166257 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:28.166229 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zp9vw" Jul 28 00:57:32.238576 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:32.238534 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:32.238980 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:32.238615 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:32.250843 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:32.250813 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:57:33.193306 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:57:33.193275 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk" Jul 28 00:58:38.416746 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.416672 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5775755d57-wbxst"] Jul 28 00:58:38.419898 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.419881 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5775755d57-wbxst" Jul 28 00:58:38.422706 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.422683 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 28 00:58:38.422805 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.422703 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gfzvw\"" Jul 28 00:58:38.428047 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.428022 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5775755d57-wbxst"] Jul 28 00:58:38.509440 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.509412 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14c48c4e-b078-4cf4-a705-ff81c9db1167-tls-cert\") pod \"authorino-5775755d57-wbxst\" (UID: \"14c48c4e-b078-4cf4-a705-ff81c9db1167\") " pod="kuadrant-system/authorino-5775755d57-wbxst" Jul 28 00:58:38.509563 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.509496 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdfhh\" (UniqueName: \"kubernetes.io/projected/14c48c4e-b078-4cf4-a705-ff81c9db1167-kube-api-access-gdfhh\") pod \"authorino-5775755d57-wbxst\" (UID: \"14c48c4e-b078-4cf4-a705-ff81c9db1167\") " pod="kuadrant-system/authorino-5775755d57-wbxst" Jul 28 00:58:38.610608 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.610586 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdfhh\" (UniqueName: \"kubernetes.io/projected/14c48c4e-b078-4cf4-a705-ff81c9db1167-kube-api-access-gdfhh\") pod \"authorino-5775755d57-wbxst\" (UID: \"14c48c4e-b078-4cf4-a705-ff81c9db1167\") " pod="kuadrant-system/authorino-5775755d57-wbxst" Jul 28 00:58:38.610705 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.610639 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14c48c4e-b078-4cf4-a705-ff81c9db1167-tls-cert\") pod \"authorino-5775755d57-wbxst\" (UID: \"14c48c4e-b078-4cf4-a705-ff81c9db1167\") " pod="kuadrant-system/authorino-5775755d57-wbxst" Jul 28 00:58:38.612866 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.612841 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/14c48c4e-b078-4cf4-a705-ff81c9db1167-tls-cert\") pod \"authorino-5775755d57-wbxst\" (UID: \"14c48c4e-b078-4cf4-a705-ff81c9db1167\") " pod="kuadrant-system/authorino-5775755d57-wbxst" Jul 28 00:58:38.619377 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.619357 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdfhh\" (UniqueName: \"kubernetes.io/projected/14c48c4e-b078-4cf4-a705-ff81c9db1167-kube-api-access-gdfhh\") pod \"authorino-5775755d57-wbxst\" (UID: \"14c48c4e-b078-4cf4-a705-ff81c9db1167\") " pod="kuadrant-system/authorino-5775755d57-wbxst" Jul 28 00:58:38.730805 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.730741 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5775755d57-wbxst" Jul 28 00:58:38.861904 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:38.861875 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5775755d57-wbxst"] Jul 28 00:58:38.863656 ip-10-0-137-173 kubenswrapper[2550]: W0728 00:58:38.863625 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14c48c4e_b078_4cf4_a705_ff81c9db1167.slice/crio-53865f46717c0176e75e6677f177cf9df639c519fcc089187a0f868a99a073d4 WatchSource:0}: Error finding container 53865f46717c0176e75e6677f177cf9df639c519fcc089187a0f868a99a073d4: Status 404 returned error can't find the container with id 53865f46717c0176e75e6677f177cf9df639c519fcc089187a0f868a99a073d4 Jul 28 00:58:39.562021 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:39.561965 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5775755d57-wbxst" event={"ID":"14c48c4e-b078-4cf4-a705-ff81c9db1167","Type":"ContainerStarted","Data":"6c9356c56a0569d2544b51a1421fd5a76d639a55021f7bbd49720e1e93cd5af6"} Jul 28 00:58:39.562408 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:39.562026 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5775755d57-wbxst" event={"ID":"14c48c4e-b078-4cf4-a705-ff81c9db1167","Type":"ContainerStarted","Data":"53865f46717c0176e75e6677f177cf9df639c519fcc089187a0f868a99a073d4"} Jul 28 00:58:39.579334 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:58:39.579283 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5775755d57-wbxst" podStartSLOduration=1.02893906 podStartE2EDuration="1.579266806s" totalImagesPullingTime="550.327746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:58:39.576165227 +0000 UTC m=+715.337013239" watchObservedRunningTime="2026-07-28 00:58:39.579266806 +0000 UTC m=+715.340114818" Jul 28 00:59:25.276774 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.276728 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-fj6fs"] Jul 28 00:59:25.277327 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.276934 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" podUID="9f2f7869-a333-41ee-8b22-f986ab0299af" containerName="manager" containerID="cri-o://3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf" gracePeriod=30 Jul 28 00:59:25.525505 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.525475 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:59:25.604586 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.604562 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f2f7869-a333-41ee-8b22-f986ab0299af-webhook-cert\") pod \"9f2f7869-a333-41ee-8b22-f986ab0299af\" (UID: \"9f2f7869-a333-41ee-8b22-f986ab0299af\") " Jul 28 00:59:25.604727 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.604613 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-258t6\" (UniqueName: \"kubernetes.io/projected/9f2f7869-a333-41ee-8b22-f986ab0299af-kube-api-access-258t6\") pod \"9f2f7869-a333-41ee-8b22-f986ab0299af\" (UID: \"9f2f7869-a333-41ee-8b22-f986ab0299af\") " Jul 28 00:59:25.606485 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.606458 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f2f7869-a333-41ee-8b22-f986ab0299af-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9f2f7869-a333-41ee-8b22-f986ab0299af" (UID: "9f2f7869-a333-41ee-8b22-f986ab0299af"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 00:59:25.606572 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.606507 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f2f7869-a333-41ee-8b22-f986ab0299af-kube-api-access-258t6" (OuterVolumeSpecName: "kube-api-access-258t6") pod "9f2f7869-a333-41ee-8b22-f986ab0299af" (UID: "9f2f7869-a333-41ee-8b22-f986ab0299af"). InnerVolumeSpecName "kube-api-access-258t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 00:59:25.705575 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.705545 2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9f2f7869-a333-41ee-8b22-f986ab0299af-webhook-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:59:25.705575 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.705573 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-258t6\" (UniqueName: \"kubernetes.io/projected/9f2f7869-a333-41ee-8b22-f986ab0299af-kube-api-access-258t6\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 00:59:25.830727 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.830698 2550 generic.go:356] "Generic (PLEG): container finished" podID="9f2f7869-a333-41ee-8b22-f986ab0299af" containerID="3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf" exitCode=0 Jul 28 00:59:25.830824 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.830752 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" Jul 28 00:59:25.830824 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.830767 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" event={"ID":"9f2f7869-a333-41ee-8b22-f986ab0299af","Type":"ContainerDied","Data":"3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf"} Jul 28 00:59:25.830900 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.830831 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-fj6fs" event={"ID":"9f2f7869-a333-41ee-8b22-f986ab0299af","Type":"ContainerDied","Data":"c1adf837cd538efff4b491270b56d2fa047c8b133e21d90509319caa6f99610e"} Jul 28 00:59:25.830900 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.830854 2550 scope.go:117] "RemoveContainer" containerID="3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf" Jul 28 00:59:25.843314 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.843294 2550 scope.go:117] "RemoveContainer" containerID="3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf" Jul 28 00:59:25.843594 ip-10-0-137-173 kubenswrapper[2550]: E0728 00:59:25.843574 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf\": container with ID starting with 3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf not found: ID does not exist" containerID="3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf" Jul 28 00:59:25.843657 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.843607 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf"} err="failed to get container status \"3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf\": rpc error: code = NotFound desc = could not find container \"3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf\": container with ID starting with 3d71261db69a8229a5d22c6b57c43ca7679e7b32aa689a861515d72c7abcafaf not found: ID does not exist" Jul 28 00:59:25.852432 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.852403 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-fj6fs"] Jul 28 00:59:25.856368 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:25.856310 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-fj6fs"] Jul 28 00:59:26.785494 ip-10-0-137-173 kubenswrapper[2550]: I0728 00:59:26.785445 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f2f7869-a333-41ee-8b22-f986ab0299af" path="/var/lib/kubelet/pods/9f2f7869-a333-41ee-8b22-f986ab0299af/volumes" Jul 28 01:00:26.636238 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.636196 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-p9jcw"] Jul 28 01:00:26.636750 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.636613 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f2f7869-a333-41ee-8b22-f986ab0299af" containerName="manager" Jul 28 01:00:26.636750 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.636626 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f2f7869-a333-41ee-8b22-f986ab0299af" containerName="manager" Jul 28 01:00:26.636750 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.636712 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f2f7869-a333-41ee-8b22-f986ab0299af" containerName="manager" Jul 28 01:00:26.639849 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.639828 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:26.642309 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.642285 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zlnnc\"" Jul 28 01:00:26.642445 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.642421 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 01:00:26.643202 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.643185 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 01:00:26.647522 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.647498 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-p9jcw"] Jul 28 01:00:26.678198 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.678171 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-webhook-cert\") pod \"maas-controller-7f6f5fc955-p9jcw\" (UID: \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\") " pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:26.678320 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.678249 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xpg\" (UniqueName: \"kubernetes.io/projected/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-kube-api-access-b2xpg\") pod \"maas-controller-7f6f5fc955-p9jcw\" (UID: \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\") " pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:26.779388 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.779350 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xpg\" (UniqueName: \"kubernetes.io/projected/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-kube-api-access-b2xpg\") pod \"maas-controller-7f6f5fc955-p9jcw\" (UID: \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\") " pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:26.779580 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.779468 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-webhook-cert\") pod \"maas-controller-7f6f5fc955-p9jcw\" (UID: \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\") " pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:26.782099 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.782068 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-webhook-cert\") pod \"maas-controller-7f6f5fc955-p9jcw\" (UID: \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\") " pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:26.787090 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.787063 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xpg\" (UniqueName: \"kubernetes.io/projected/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-kube-api-access-b2xpg\") pod \"maas-controller-7f6f5fc955-p9jcw\" (UID: \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\") " pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:26.952942 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:26.952870 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:27.081381 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:27.081353 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-p9jcw"] Jul 28 01:00:27.083330 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:00:27.083306 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9831e80d_77b3_4f9b_ac97_a6e1f47a011b.slice/crio-8fca32ec6b252d503b43e88f14bccc503b77e73c57f520b266e8baf5a510b1be WatchSource:0}: Error finding container 8fca32ec6b252d503b43e88f14bccc503b77e73c57f520b266e8baf5a510b1be: Status 404 returned error can't find the container with id 8fca32ec6b252d503b43e88f14bccc503b77e73c57f520b266e8baf5a510b1be Jul 28 01:00:27.188653 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:27.188623 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" event={"ID":"9831e80d-77b3-4f9b-ac97-a6e1f47a011b","Type":"ContainerStarted","Data":"8fca32ec6b252d503b43e88f14bccc503b77e73c57f520b266e8baf5a510b1be"} Jul 28 01:00:28.196079 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:28.196038 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" event={"ID":"9831e80d-77b3-4f9b-ac97-a6e1f47a011b","Type":"ContainerStarted","Data":"398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef"} Jul 28 01:00:28.196443 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:28.196102 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:28.212687 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:28.212639 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" podStartSLOduration=1.71530672 podStartE2EDuration="2.212622918s" totalImagesPullingTime="497.316198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:00:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:00:28.211461049 +0000 UTC m=+823.972309063" watchObservedRunningTime="2026-07-28 01:00:28.212622918 +0000 UTC m=+823.973470930" Jul 28 01:00:39.205428 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:39.205398 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:43.860602 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:43.860565 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-p9jcw"] Jul 28 01:00:43.861104 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:43.860899 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" podUID="9831e80d-77b3-4f9b-ac97-a6e1f47a011b" containerName="manager" containerID="cri-o://398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef" gracePeriod=30 Jul 28 01:00:44.108585 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.108556 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:44.218874 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.218809 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-webhook-cert\") pod \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\" (UID: \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\") " Jul 28 01:00:44.218874 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.218843 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2xpg\" (UniqueName: \"kubernetes.io/projected/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-kube-api-access-b2xpg\") pod \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\" (UID: \"9831e80d-77b3-4f9b-ac97-a6e1f47a011b\") " Jul 28 01:00:44.220756 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.220719 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9831e80d-77b3-4f9b-ac97-a6e1f47a011b" (UID: "9831e80d-77b3-4f9b-ac97-a6e1f47a011b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:00:44.220875 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.220822 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-kube-api-access-b2xpg" (OuterVolumeSpecName: "kube-api-access-b2xpg") pod "9831e80d-77b3-4f9b-ac97-a6e1f47a011b" (UID: "9831e80d-77b3-4f9b-ac97-a6e1f47a011b"). InnerVolumeSpecName "kube-api-access-b2xpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:00:44.289792 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.289762 2550 generic.go:356] "Generic (PLEG): container finished" podID="9831e80d-77b3-4f9b-ac97-a6e1f47a011b" containerID="398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef" exitCode=0 Jul 28 01:00:44.289904 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.289806 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" Jul 28 01:00:44.289904 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.289852 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" event={"ID":"9831e80d-77b3-4f9b-ac97-a6e1f47a011b","Type":"ContainerDied","Data":"398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef"} Jul 28 01:00:44.289977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.289905 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-p9jcw" event={"ID":"9831e80d-77b3-4f9b-ac97-a6e1f47a011b","Type":"ContainerDied","Data":"8fca32ec6b252d503b43e88f14bccc503b77e73c57f520b266e8baf5a510b1be"} Jul 28 01:00:44.289977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.289928 2550 scope.go:117] "RemoveContainer" containerID="398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef" Jul 28 01:00:44.299153 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.299138 2550 scope.go:117] "RemoveContainer" containerID="398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef" Jul 28 01:00:44.299374 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:00:44.299357 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef\": container with ID starting with 398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef not found: ID does not exist" containerID="398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef" Jul 28 01:00:44.299420 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.299384 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef"} err="failed to get container status \"398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef\": rpc error: code = NotFound desc = could not find container \"398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef\": container with ID starting with 398d2ca34fb5a1e30628716a25580e2bc8d2a465988d18bab3cc357c1dc104ef not found: ID does not exist" Jul 28 01:00:44.311359 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.311335 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-p9jcw"] Jul 28 01:00:44.314900 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.314866 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-p9jcw"] Jul 28 01:00:44.319800 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.319782 2550 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-webhook-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:00:44.319855 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.319803 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xpg\" (UniqueName: \"kubernetes.io/projected/9831e80d-77b3-4f9b-ac97-a6e1f47a011b-kube-api-access-b2xpg\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:00:44.785103 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:44.785074 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9831e80d-77b3-4f9b-ac97-a6e1f47a011b" path="/var/lib/kubelet/pods/9831e80d-77b3-4f9b-ac97-a6e1f47a011b/volumes" Jul 28 01:00:45.051295 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.051267 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-dvvj7"] Jul 28 01:00:45.051649 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.051639 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9831e80d-77b3-4f9b-ac97-a6e1f47a011b" containerName="manager" Jul 28 01:00:45.051649 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.051649 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="9831e80d-77b3-4f9b-ac97-a6e1f47a011b" containerName="manager" Jul 28 01:00:45.051727 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.051711 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="9831e80d-77b3-4f9b-ac97-a6e1f47a011b" containerName="manager" Jul 28 01:00:45.056036 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.056001 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:00:45.059157 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.059135 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 28 01:00:45.059295 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.059156 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 28 01:00:45.059295 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.059219 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zlnnc\"" Jul 28 01:00:45.063653 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.063458 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-dvvj7"] Jul 28 01:00:45.124507 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.124478 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d48ab8c7-f5de-4a90-b371-ca85c21aa1a8-webhook-cert\") pod \"maas-controller-7f6f5fc955-dvvj7\" (UID: \"d48ab8c7-f5de-4a90-b371-ca85c21aa1a8\") " pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:00:45.124616 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.124520 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f66b\" (UniqueName: \"kubernetes.io/projected/d48ab8c7-f5de-4a90-b371-ca85c21aa1a8-kube-api-access-5f66b\") pod \"maas-controller-7f6f5fc955-dvvj7\" (UID: \"d48ab8c7-f5de-4a90-b371-ca85c21aa1a8\") " pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:00:45.224966 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.224945 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d48ab8c7-f5de-4a90-b371-ca85c21aa1a8-webhook-cert\") pod \"maas-controller-7f6f5fc955-dvvj7\" (UID: \"d48ab8c7-f5de-4a90-b371-ca85c21aa1a8\") " pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:00:45.225078 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.224985 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f66b\" (UniqueName: \"kubernetes.io/projected/d48ab8c7-f5de-4a90-b371-ca85c21aa1a8-kube-api-access-5f66b\") pod \"maas-controller-7f6f5fc955-dvvj7\" (UID: \"d48ab8c7-f5de-4a90-b371-ca85c21aa1a8\") " pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:00:45.227169 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.227148 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d48ab8c7-f5de-4a90-b371-ca85c21aa1a8-webhook-cert\") pod \"maas-controller-7f6f5fc955-dvvj7\" (UID: \"d48ab8c7-f5de-4a90-b371-ca85c21aa1a8\") " pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:00:45.232539 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.232518 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f66b\" (UniqueName: \"kubernetes.io/projected/d48ab8c7-f5de-4a90-b371-ca85c21aa1a8-kube-api-access-5f66b\") pod \"maas-controller-7f6f5fc955-dvvj7\" (UID: \"d48ab8c7-f5de-4a90-b371-ca85c21aa1a8\") " pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:00:45.367797 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.367742 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:00:45.497329 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:45.497291 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7f6f5fc955-dvvj7"] Jul 28 01:00:45.500116 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:00:45.500089 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd48ab8c7_f5de_4a90_b371_ca85c21aa1a8.slice/crio-3fb91067dd4bafd367b5a36cc896b140c9afec449aaeaf20860f6c3d27ad0c6f WatchSource:0}: Error finding container 3fb91067dd4bafd367b5a36cc896b140c9afec449aaeaf20860f6c3d27ad0c6f: Status 404 returned error can't find the container with id 3fb91067dd4bafd367b5a36cc896b140c9afec449aaeaf20860f6c3d27ad0c6f Jul 28 01:00:46.303669 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:46.303636 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" event={"ID":"d48ab8c7-f5de-4a90-b371-ca85c21aa1a8","Type":"ContainerStarted","Data":"d810ac6a63f9f5ebb2d7327e9a91a1923739fd928d4d04c60ddb67243dda80ec"} Jul 28 01:00:46.303988 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:46.303697 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" event={"ID":"d48ab8c7-f5de-4a90-b371-ca85c21aa1a8","Type":"ContainerStarted","Data":"3fb91067dd4bafd367b5a36cc896b140c9afec449aaeaf20860f6c3d27ad0c6f"} Jul 28 01:00:46.303988 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:46.303719 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:00:46.321052 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:46.320987 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" podStartSLOduration=0.636168694 podStartE2EDuration="1.320966767s" totalImagesPullingTime="684.798073ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:00:46.319069542 +0000 UTC m=+842.079917558" watchObservedRunningTime="2026-07-28 01:00:46.320966767 +0000 UTC m=+842.081814780" Jul 28 01:00:57.311536 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:00:57.311502 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7f6f5fc955-dvvj7" Jul 28 01:09:27.509870 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.509836 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7"] Jul 28 01:09:27.513766 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.513741 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.516075 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.516052 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-fa8a1ecc-kserve-self-signed-certs\"" Jul 28 01:09:27.521298 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.521250 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7"] Jul 28 01:09:27.637859 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.637826 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-model-cache\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.637859 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.637859 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tls-certs\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.638079 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.637900 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tmp-dir\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.638079 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.637986 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-home\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.638079 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.638065 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-dshm\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.638185 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.638134 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmxwg\" (UniqueName: \"kubernetes.io/projected/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-kube-api-access-bmxwg\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.738817 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.738784 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-home\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.738817 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.738817 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-dshm\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.739057 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.738855 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmxwg\" (UniqueName: \"kubernetes.io/projected/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-kube-api-access-bmxwg\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.739057 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.738890 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-model-cache\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.739057 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.738906 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tls-certs\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.739057 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.738948 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tmp-dir\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.739349 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.739325 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-home\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.739402 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.739344 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tmp-dir\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.739402 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.739391 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-model-cache\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.741073 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.741050 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-dshm\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.741469 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.741444 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tls-certs\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.746605 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.746586 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmxwg\" (UniqueName: \"kubernetes.io/projected/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-kube-api-access-bmxwg\") pod \"e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.824669 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.824640 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:27.966076 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:27.966048 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7"] Jul 28 01:09:27.967423 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:09:27.967391 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad769ec8_af40_4f3d_aaf7_2d8cad060b8b.slice/crio-0e476321175eecea86e9e9025644bb71ca65a289cda577c5835c0669501370b8 WatchSource:0}: Error finding container 0e476321175eecea86e9e9025644bb71ca65a289cda577c5835c0669501370b8: Status 404 returned error can't find the container with id 0e476321175eecea86e9e9025644bb71ca65a289cda577c5835c0669501370b8 Jul 28 01:09:28.322817 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:28.322775 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" event={"ID":"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b","Type":"ContainerStarted","Data":"74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4"} Jul 28 01:09:28.323021 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:28.322829 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" event={"ID":"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b","Type":"ContainerStarted","Data":"0e476321175eecea86e9e9025644bb71ca65a289cda577c5835c0669501370b8"} Jul 28 01:09:28.339091 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:28.338999 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" podStartSLOduration=1.338979334 podStartE2EDuration="1.338979334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:09:28.338631994 +0000 UTC m=+1364.099480008" watchObservedRunningTime="2026-07-28 01:09:28.338979334 +0000 UTC m=+1364.099827348" Jul 28 01:09:37.824757 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:37.824721 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:37.824757 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:37.824762 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:37.837049 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:37.837026 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:09:38.395722 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:09:38.395690 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:10:01.192161 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:01.192129 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7"] Jul 28 01:10:16.213322 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.213273 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" podUID="ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" containerName="main" containerID="cri-o://74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4" gracePeriod=45 Jul 28 01:10:16.466945 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.466892 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:10:16.535960 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.535932 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-home\") pod \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " Jul 28 01:10:16.536108 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536022 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmxwg\" (UniqueName: \"kubernetes.io/projected/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-kube-api-access-bmxwg\") pod \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " Jul 28 01:10:16.536108 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536054 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-dshm\") pod \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " Jul 28 01:10:16.536225 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536135 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tmp-dir\") pod \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " Jul 28 01:10:16.536225 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536170 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-model-cache\") pod \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " Jul 28 01:10:16.536225 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536185 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-home" (OuterVolumeSpecName: "home") pod "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" (UID: "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:10:16.536225 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536215 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tls-certs\") pod \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\" (UID: \"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b\") " Jul 28 01:10:16.536417 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536377 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" (UID: "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:10:16.536417 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536390 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" (UID: "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:10:16.536522 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536506 2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:10:16.536522 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536520 2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:10:16.536613 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.536529 2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:10:16.538130 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.538104 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-dshm" (OuterVolumeSpecName: "dshm") pod "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" (UID: "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:10:16.538227 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.538181 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-kube-api-access-bmxwg" (OuterVolumeSpecName: "kube-api-access-bmxwg") pod "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" (UID: "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b"). InnerVolumeSpecName "kube-api-access-bmxwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:10:16.538227 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.538191 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" (UID: "ad769ec8-af40-4f3d-aaf7-2d8cad060b8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:10:16.612425 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.612398 2550 generic.go:356] "Generic (PLEG): container finished" podID="ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" containerID="74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4" exitCode=0 Jul 28 01:10:16.612517 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.612443 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" Jul 28 01:10:16.612517 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.612478 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" event={"ID":"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b","Type":"ContainerDied","Data":"74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4"} Jul 28 01:10:16.612517 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.612516 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7" event={"ID":"ad769ec8-af40-4f3d-aaf7-2d8cad060b8b","Type":"ContainerDied","Data":"0e476321175eecea86e9e9025644bb71ca65a289cda577c5835c0669501370b8"} Jul 28 01:10:16.612682 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.612534 2550 scope.go:117] "RemoveContainer" containerID="74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4" Jul 28 01:10:16.627602 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.627585 2550 scope.go:117] "RemoveContainer" containerID="74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4" Jul 28 01:10:16.627839 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:10:16.627820 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4\": container with ID starting with 74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4 not found: ID does not exist" containerID="74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4" Jul 28 01:10:16.627903 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.627851 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4"} err="failed to get container status \"74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4\": rpc error: code = NotFound desc = could not find container \"74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4\": container with ID starting with 74ebe5bf005161d56f18c8a73d814238488d778f6cf565bf5d932950708332e4 not found: ID does not exist" Jul 28 01:10:16.637783 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.637763 2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:10:16.637783 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.637783 2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:10:16.637940 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.637793 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmxwg\" (UniqueName: \"kubernetes.io/projected/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b-kube-api-access-bmxwg\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:10:16.638027 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.637992 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7"] Jul 28 01:10:16.643317 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.643297 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-fa8a1ecc-kserve-6bcbb86467-kmhg7"] Jul 28 01:10:16.784798 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:10:16.784740 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" path="/var/lib/kubelet/pods/ad769ec8-af40-4f3d-aaf7-2d8cad060b8b/volumes" Jul 28 01:13:52.674522 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.674453 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5"] Jul 28 01:13:52.676560 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.674838 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" containerName="main" Jul 28 01:13:52.676560 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.674849 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" containerName="main" Jul 28 01:13:52.676560 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.674923 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad769ec8-af40-4f3d-aaf7-2d8cad060b8b" containerName="main" Jul 28 01:13:52.677878 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.677862 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.680142 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.680123 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1dcfab82-openshift-default-dockercfg-hfjpm\"" Jul 28 01:13:52.698196 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.698173 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5"] Jul 28 01:13:52.786316 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.786281 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-token\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.786503 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.786330 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55973ab1-eb51-45f4-bc67-418ef358d7a0-istiod-ca-cert\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.786503 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.786391 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-socket\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.786503 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.786436 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l2hn\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-kube-api-access-7l2hn\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.786503 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.786468 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-envoy\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.786724 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.786516 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-podinfo\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.786724 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.786545 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-data\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.786724 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.786606 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-certs\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.786724 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.786636 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-credential-socket\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.887766 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.887733 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-data\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.887914 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.887783 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-certs\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.887914 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.887801 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-credential-socket\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.887996 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.887909 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-token\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.887996 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.887942 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55973ab1-eb51-45f4-bc67-418ef358d7a0-istiod-ca-cert\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.887996 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.887990 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-socket\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.888162 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.888046 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l2hn\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-kube-api-access-7l2hn\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.888162 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.888074 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-envoy\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.888162 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.888123 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-podinfo\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.888162 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.888148 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-certs\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.888374 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.888344 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-credential-socket\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.888420 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.888366 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-data\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.888508 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.888485 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-socket\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.888859 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.888841 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55973ab1-eb51-45f4-bc67-418ef358d7a0-istiod-ca-cert\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.890379 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.890351 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-envoy\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.890663 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.890645 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-podinfo\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.895690 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.895669 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-token\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.895886 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.895863 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l2hn\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-kube-api-access-7l2hn\") pod \"e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:52.987695 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:52.987632 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:53.123927 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:53.123899 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5"] Jul 28 01:13:53.125609 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:13:53.125583 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55973ab1_eb51_45f4_bc67_418ef358d7a0.slice/crio-e3022635f2b7fbb8a83ed50ecc2af67eade9472d370e46e3eadec0d34d2e95d3 WatchSource:0}: Error finding container e3022635f2b7fbb8a83ed50ecc2af67eade9472d370e46e3eadec0d34d2e95d3: Status 404 returned error can't find the container with id e3022635f2b7fbb8a83ed50ecc2af67eade9472d370e46e3eadec0d34d2e95d3 Jul 28 01:13:53.127900 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:53.127863 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:13:53.128030 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:53.127947 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:13:53.128030 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:53.127993 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:13:53.869095 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:53.869060 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" event={"ID":"55973ab1-eb51-45f4-bc67-418ef358d7a0","Type":"ContainerStarted","Data":"ae8e1c86972540452f27ef00b1db29b7ffa1ea04f6d17e840d5cbba0e59470f2"} Jul 28 01:13:53.869490 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:53.869119 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" event={"ID":"55973ab1-eb51-45f4-bc67-418ef358d7a0","Type":"ContainerStarted","Data":"e3022635f2b7fbb8a83ed50ecc2af67eade9472d370e46e3eadec0d34d2e95d3"} Jul 28 01:13:53.889953 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:53.889905 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" podStartSLOduration=1.8898900159999998 podStartE2EDuration="1.889890016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:13:53.888832496 +0000 UTC m=+1629.649680508" watchObservedRunningTime="2026-07-28 01:13:53.889890016 +0000 UTC m=+1629.650738029" Jul 28 01:13:53.988267 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:53.988243 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:54.992975 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:54.992937 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:55.880454 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:55.880424 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:13:55.881190 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:13:55.881169 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:14:10.935638 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:10.935465 2550 patch_prober.go:28] interesting pod/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:43380->10.134.0.53:15021: read: connection reset by peer" start-of-body= Jul 28 01:14:10.935638 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:10.935529 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:43380->10.134.0.53:15021: read: connection reset by peer" Jul 28 01:14:10.969428 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:10.969406 2550 generic.go:356] "Generic (PLEG): container finished" podID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerID="ae8e1c86972540452f27ef00b1db29b7ffa1ea04f6d17e840d5cbba0e59470f2" exitCode=0 Jul 28 01:14:10.969527 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:10.969459 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" event={"ID":"55973ab1-eb51-45f4-bc67-418ef358d7a0","Type":"ContainerDied","Data":"ae8e1c86972540452f27ef00b1db29b7ffa1ea04f6d17e840d5cbba0e59470f2"} Jul 28 01:14:10.969806 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:10.969793 2550 scope.go:117] "RemoveContainer" containerID="ae8e1c86972540452f27ef00b1db29b7ffa1ea04f6d17e840d5cbba0e59470f2" Jul 28 01:14:10.970784 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:10.970756 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:14:10.970842 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:10.970828 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:14:10.970878 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:10.970865 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:14:11.978020 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:11.977982 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" event={"ID":"55973ab1-eb51-45f4-bc67-418ef358d7a0","Type":"ContainerStarted","Data":"3360c273359462a7f051737b9ba5a09e8d9f8e4204104efa3fa63df30190f6ec"} Jul 28 01:14:11.988147 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:11.988125 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:14:12.988744 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:12.988702 2550 patch_prober.go:28] interesting pod/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 28 01:14:12.989205 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:12.988771 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 28 01:14:13.988170 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:13.988135 2550 patch_prober.go:28] interesting pod/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 28 01:14:13.988357 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:13.988204 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 28 01:14:14.988945 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:14.988911 2550 patch_prober.go:28] interesting pod/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 28 01:14:14.989363 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:14.988979 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 28 01:14:15.991541 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:15.991512 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:14:15.991945 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:15.991751 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:14:15.992432 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:15.992415 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:14:35.947823 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:35.947783 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5"] Jul 28 01:14:35.948408 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:35.948051 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" containerID="cri-o://3360c273359462a7f051737b9ba5a09e8d9f8e4204104efa3fa63df30190f6ec" gracePeriod=30 Jul 28 01:14:41.144908 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.144875 2550 generic.go:356] "Generic (PLEG): container finished" podID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerID="3360c273359462a7f051737b9ba5a09e8d9f8e4204104efa3fa63df30190f6ec" exitCode=0 Jul 28 01:14:41.145298 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.144942 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" event={"ID":"55973ab1-eb51-45f4-bc67-418ef358d7a0","Type":"ContainerDied","Data":"3360c273359462a7f051737b9ba5a09e8d9f8e4204104efa3fa63df30190f6ec"} Jul 28 01:14:41.145298 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.145019 2550 scope.go:117] "RemoveContainer" containerID="ae8e1c86972540452f27ef00b1db29b7ffa1ea04f6d17e840d5cbba0e59470f2" Jul 28 01:14:41.212040 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.211962 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:14:41.307038 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.306994 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-podinfo\") pod \"55973ab1-eb51-45f4-bc67-418ef358d7a0\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " Jul 28 01:14:41.307182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307053 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-token\") pod \"55973ab1-eb51-45f4-bc67-418ef358d7a0\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " Jul 28 01:14:41.307182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307094 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l2hn\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-kube-api-access-7l2hn\") pod \"55973ab1-eb51-45f4-bc67-418ef358d7a0\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " Jul 28 01:14:41.307182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307116 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-credential-socket\") pod \"55973ab1-eb51-45f4-bc67-418ef358d7a0\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " Jul 28 01:14:41.307182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307171 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55973ab1-eb51-45f4-bc67-418ef358d7a0-istiod-ca-cert\") pod \"55973ab1-eb51-45f4-bc67-418ef358d7a0\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " Jul 28 01:14:41.307400 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307198 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-socket\") pod \"55973ab1-eb51-45f4-bc67-418ef358d7a0\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " Jul 28 01:14:41.307400 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307229 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-envoy\") pod \"55973ab1-eb51-45f4-bc67-418ef358d7a0\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " Jul 28 01:14:41.307400 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307271 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-data\") pod \"55973ab1-eb51-45f4-bc67-418ef358d7a0\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " Jul 28 01:14:41.307400 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307310 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-certs\") pod \"55973ab1-eb51-45f4-bc67-418ef358d7a0\" (UID: \"55973ab1-eb51-45f4-bc67-418ef358d7a0\") " Jul 28 01:14:41.307613 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307466 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "55973ab1-eb51-45f4-bc67-418ef358d7a0" (UID: "55973ab1-eb51-45f4-bc67-418ef358d7a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:14:41.307613 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307478 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "55973ab1-eb51-45f4-bc67-418ef358d7a0" (UID: "55973ab1-eb51-45f4-bc67-418ef358d7a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:14:41.307613 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307564 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55973ab1-eb51-45f4-bc67-418ef358d7a0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "55973ab1-eb51-45f4-bc67-418ef358d7a0" (UID: "55973ab1-eb51-45f4-bc67-418ef358d7a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:14:41.307771 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307674 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "55973ab1-eb51-45f4-bc67-418ef358d7a0" (UID: "55973ab1-eb51-45f4-bc67-418ef358d7a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:14:41.307771 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307748 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:14:41.307771 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307765 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:14:41.307930 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307779 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55973ab1-eb51-45f4-bc67-418ef358d7a0-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:14:41.307930 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.307793 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:14:41.308179 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.308161 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-data" (OuterVolumeSpecName: "istio-data") pod "55973ab1-eb51-45f4-bc67-418ef358d7a0" (UID: "55973ab1-eb51-45f4-bc67-418ef358d7a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:14:41.309405 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.309384 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "55973ab1-eb51-45f4-bc67-418ef358d7a0" (UID: "55973ab1-eb51-45f4-bc67-418ef358d7a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 01:14:41.309489 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.309442 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "55973ab1-eb51-45f4-bc67-418ef358d7a0" (UID: "55973ab1-eb51-45f4-bc67-418ef358d7a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:14:41.309587 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.309571 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-kube-api-access-7l2hn" (OuterVolumeSpecName: "kube-api-access-7l2hn") pod "55973ab1-eb51-45f4-bc67-418ef358d7a0" (UID: "55973ab1-eb51-45f4-bc67-418ef358d7a0"). InnerVolumeSpecName "kube-api-access-7l2hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:14:41.309640 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.309588 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-token" (OuterVolumeSpecName: "istio-token") pod "55973ab1-eb51-45f4-bc67-418ef358d7a0" (UID: "55973ab1-eb51-45f4-bc67-418ef358d7a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:14:41.408699 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.408674 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:14:41.408699 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.408697 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:14:41.408830 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.408710 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l2hn\" (UniqueName: \"kubernetes.io/projected/55973ab1-eb51-45f4-bc67-418ef358d7a0-kube-api-access-7l2hn\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:14:41.408830 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.408723 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:14:41.408830 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:41.408734 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55973ab1-eb51-45f4-bc67-418ef358d7a0-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:14:42.152046 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:42.152020 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" Jul 28 01:14:42.152489 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:42.152045 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5" event={"ID":"55973ab1-eb51-45f4-bc67-418ef358d7a0","Type":"ContainerDied","Data":"e3022635f2b7fbb8a83ed50ecc2af67eade9472d370e46e3eadec0d34d2e95d3"} Jul 28 01:14:42.152489 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:42.152108 2550 scope.go:117] "RemoveContainer" containerID="3360c273359462a7f051737b9ba5a09e8d9f8e4204104efa3fa63df30190f6ec" Jul 28 01:14:42.181506 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:42.181470 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5"] Jul 28 01:14:42.182800 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:42.182782 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1dcfab82-openshift-default-64d7dc578b-kv5b5"] Jul 28 01:14:42.785406 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:14:42.785373 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" path="/var/lib/kubelet/pods/55973ab1-eb51-45f4-bc67-418ef358d7a0/volumes" Jul 28 01:15:23.454963 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.454926 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b"] Jul 28 01:15:23.455478 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.455452 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" Jul 28 01:15:23.455478 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.455471 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" Jul 28 01:15:23.455563 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.455487 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" Jul 28 01:15:23.455563 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.455495 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" Jul 28 01:15:23.455626 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.455567 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" Jul 28 01:15:23.455626 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.455575 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="55973ab1-eb51-45f4-bc67-418ef358d7a0" containerName="istio-proxy" Jul 28 01:15:23.458597 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.458577 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.461687 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.461669 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-60243cce-openshift-default-dockercfg-8vt5b\"" Jul 28 01:15:23.469093 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.469065 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b"] Jul 28 01:15:23.550821 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.550786 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-certs\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.550950 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.550828 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-envoy\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.550950 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.550857 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-data\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.551069 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.550951 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-credential-socket\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.551069 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.550999 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-socket\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.551069 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.551059 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d271be-9806-4e8f-9787-10f7140549f6-istio-podinfo\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.551187 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.551119 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d271be-9806-4e8f-9787-10f7140549f6-istiod-ca-cert\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.551187 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.551146 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xnc2\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-kube-api-access-4xnc2\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.551267 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.551193 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-istio-token\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652157 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652123 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-certs\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652375 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652351 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-envoy\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652469 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652413 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-data\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652523 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652475 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-credential-socket\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652523 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652512 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-socket\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652627 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652538 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-certs\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652627 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652554 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d271be-9806-4e8f-9787-10f7140549f6-istio-podinfo\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652748 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652690 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d271be-9806-4e8f-9787-10f7140549f6-istiod-ca-cert\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652748 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652728 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnc2\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-kube-api-access-4xnc2\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652858 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652774 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-istio-token\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652858 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652786 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-data\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652876 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-credential-socket\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.652977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.652953 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-socket\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.653370 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.653348 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d271be-9806-4e8f-9787-10f7140549f6-istiod-ca-cert\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.654590 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.654571 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-envoy\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.654888 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.654870 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d271be-9806-4e8f-9787-10f7140549f6-istio-podinfo\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.660144 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.660120 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xnc2\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-kube-api-access-4xnc2\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.660255 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.660236 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-istio-token\") pod \"e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.769139 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.769083 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:23.905169 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.905131 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b"] Jul 28 01:15:23.912460 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:15:23.912428 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d271be_9806_4e8f_9787_10f7140549f6.slice/crio-32fe526bf65acb4987b4789f4e3c9f25e4690527ddbf957828b35a9760bfdf65 WatchSource:0}: Error finding container 32fe526bf65acb4987b4789f4e3c9f25e4690527ddbf957828b35a9760bfdf65: Status 404 returned error can't find the container with id 32fe526bf65acb4987b4789f4e3c9f25e4690527ddbf957828b35a9760bfdf65 Jul 28 01:15:23.914530 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.914496 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:15:23.914622 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.914579 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:15:23.914622 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:23.914606 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:15:24.402343 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:24.402298 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" event={"ID":"13d271be-9806-4e8f-9787-10f7140549f6","Type":"ContainerStarted","Data":"32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5"} Jul 28 01:15:24.402536 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:24.402353 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" event={"ID":"13d271be-9806-4e8f-9787-10f7140549f6","Type":"ContainerStarted","Data":"32fe526bf65acb4987b4789f4e3c9f25e4690527ddbf957828b35a9760bfdf65"} Jul 28 01:15:24.425319 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:24.425276 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" podStartSLOduration=1.42526173 podStartE2EDuration="1.42526173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:15:24.424081709 +0000 UTC m=+1720.184929724" watchObservedRunningTime="2026-07-28 01:15:24.42526173 +0000 UTC m=+1720.186109798" Jul 28 01:15:24.769446 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:24.769378 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:24.773852 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:24.773832 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:25.408856 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:25.408827 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:25.409818 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:25.409799 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:40.410658 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:40.410621 2550 patch_prober.go:28] interesting pod/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 28 01:15:40.411110 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:40.410696 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 28 01:15:40.507050 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:40.506998 2550 generic.go:356] "Generic (PLEG): container finished" podID="13d271be-9806-4e8f-9787-10f7140549f6" containerID="32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5" exitCode=0 Jul 28 01:15:40.507185 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:40.507079 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" event={"ID":"13d271be-9806-4e8f-9787-10f7140549f6","Type":"ContainerDied","Data":"32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5"} Jul 28 01:15:40.507498 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:40.507474 2550 scope.go:117] "RemoveContainer" containerID="32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5" Jul 28 01:15:40.508540 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:40.508512 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:15:40.508601 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:40.508586 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:15:40.508641 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:40.508623 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:15:41.520841 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:41.520805 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" event={"ID":"13d271be-9806-4e8f-9787-10f7140549f6","Type":"ContainerStarted","Data":"5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6"} Jul 28 01:15:41.770175 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:41.770134 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:41.770463 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:41.770439 2550 patch_prober.go:28] interesting pod/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 28 01:15:41.770626 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:41.770599 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 28 01:15:42.769744 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:42.769703 2550 patch_prober.go:28] interesting pod/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 28 01:15:42.770249 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:42.769781 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 28 01:15:43.770042 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:43.769980 2550 patch_prober.go:28] interesting pod/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 28 01:15:43.770506 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:43.770072 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 28 01:15:44.769440 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:44.769403 2550 patch_prober.go:28] interesting pod/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 28 01:15:44.769716 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:44.769687 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 28 01:15:45.773509 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:45.773478 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:45.773904 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:45.773882 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:15:45.774507 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:15:45.774489 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:16:07.774188 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:07.774150 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b"] Jul 28 01:16:07.774623 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:07.774383 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" containerID="cri-o://5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6" gracePeriod=30 Jul 28 01:16:13.016991 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.016967 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:16:13.185957 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.185921 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-certs\") pod \"13d271be-9806-4e8f-9787-10f7140549f6\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " Jul 28 01:16:13.186153 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.185967 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d271be-9806-4e8f-9787-10f7140549f6-istio-podinfo\") pod \"13d271be-9806-4e8f-9787-10f7140549f6\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " Jul 28 01:16:13.186153 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.185993 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-credential-socket\") pod \"13d271be-9806-4e8f-9787-10f7140549f6\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " Jul 28 01:16:13.186153 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.186032 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xnc2\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-kube-api-access-4xnc2\") pod \"13d271be-9806-4e8f-9787-10f7140549f6\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " Jul 28 01:16:13.186153 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.186062 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-istio-token\") pod \"13d271be-9806-4e8f-9787-10f7140549f6\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " Jul 28 01:16:13.186153 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.186105 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-data\") pod \"13d271be-9806-4e8f-9787-10f7140549f6\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " Jul 28 01:16:13.186153 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.186127 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-envoy\") pod \"13d271be-9806-4e8f-9787-10f7140549f6\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " Jul 28 01:16:13.186498 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.186178 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d271be-9806-4e8f-9787-10f7140549f6-istiod-ca-cert\") pod \"13d271be-9806-4e8f-9787-10f7140549f6\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " Jul 28 01:16:13.186498 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.186218 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-socket\") pod \"13d271be-9806-4e8f-9787-10f7140549f6\" (UID: \"13d271be-9806-4e8f-9787-10f7140549f6\") " Jul 28 01:16:13.186498 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.186240 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "13d271be-9806-4e8f-9787-10f7140549f6" (UID: "13d271be-9806-4e8f-9787-10f7140549f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:16:13.186498 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.186382 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "13d271be-9806-4e8f-9787-10f7140549f6" (UID: "13d271be-9806-4e8f-9787-10f7140549f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:16:13.186717 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.186688 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d271be-9806-4e8f-9787-10f7140549f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "13d271be-9806-4e8f-9787-10f7140549f6" (UID: "13d271be-9806-4e8f-9787-10f7140549f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:16:13.187131 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.187086 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "13d271be-9806-4e8f-9787-10f7140549f6" (UID: "13d271be-9806-4e8f-9787-10f7140549f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:16:13.187246 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.187173 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "13d271be-9806-4e8f-9787-10f7140549f6" (UID: "13d271be-9806-4e8f-9787-10f7140549f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:16:13.187313 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.187275 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:16:13.187313 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.187293 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:16:13.187313 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.187308 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:16:13.187475 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.187321 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d271be-9806-4e8f-9787-10f7140549f6-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:16:13.187475 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.187333 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:16:13.188393 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.188365 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "13d271be-9806-4e8f-9787-10f7140549f6" (UID: "13d271be-9806-4e8f-9787-10f7140549f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:16:13.188577 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.188549 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-kube-api-access-4xnc2" (OuterVolumeSpecName: "kube-api-access-4xnc2") pod "13d271be-9806-4e8f-9787-10f7140549f6" (UID: "13d271be-9806-4e8f-9787-10f7140549f6"). InnerVolumeSpecName "kube-api-access-4xnc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:16:13.188674 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.188595 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "13d271be-9806-4e8f-9787-10f7140549f6" (UID: "13d271be-9806-4e8f-9787-10f7140549f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:16:13.188674 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.188604 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13d271be-9806-4e8f-9787-10f7140549f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "13d271be-9806-4e8f-9787-10f7140549f6" (UID: "13d271be-9806-4e8f-9787-10f7140549f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 01:16:13.288667 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.288641 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d271be-9806-4e8f-9787-10f7140549f6-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:16:13.288667 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.288662 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xnc2\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-kube-api-access-4xnc2\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:16:13.288810 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.288673 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d271be-9806-4e8f-9787-10f7140549f6-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:16:13.288810 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.288681 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d271be-9806-4e8f-9787-10f7140549f6-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:16:13.710791 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.710756 2550 generic.go:356] "Generic (PLEG): container finished" podID="13d271be-9806-4e8f-9787-10f7140549f6" containerID="5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6" exitCode=0 Jul 28 01:16:13.710962 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.710802 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" Jul 28 01:16:13.710962 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.710838 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" event={"ID":"13d271be-9806-4e8f-9787-10f7140549f6","Type":"ContainerDied","Data":"5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6"} Jul 28 01:16:13.710962 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.710901 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b" event={"ID":"13d271be-9806-4e8f-9787-10f7140549f6","Type":"ContainerDied","Data":"32fe526bf65acb4987b4789f4e3c9f25e4690527ddbf957828b35a9760bfdf65"} Jul 28 01:16:13.710962 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.710938 2550 scope.go:117] "RemoveContainer" containerID="5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6" Jul 28 01:16:13.720788 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.720770 2550 scope.go:117] "RemoveContainer" containerID="32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5" Jul 28 01:16:13.728838 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.728815 2550 scope.go:117] "RemoveContainer" containerID="5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6" Jul 28 01:16:13.729158 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:16:13.729134 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6\": container with ID starting with 5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6 not found: ID does not exist" containerID="5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6" Jul 28 01:16:13.729328 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.729186 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6"} err="failed to get container status \"5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6\": rpc error: code = NotFound desc = could not find container \"5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6\": container with ID starting with 5b010e562c85a6415bb5d226903d4f2cae8d29cd3bb4f9a5653e22ca6f3ee2b6 not found: ID does not exist" Jul 28 01:16:13.729328 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.729230 2550 scope.go:117] "RemoveContainer" containerID="32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5" Jul 28 01:16:13.729610 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:16:13.729583 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5\": container with ID starting with 32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5 not found: ID does not exist" containerID="32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5" Jul 28 01:16:13.729727 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.729616 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5"} err="failed to get container status \"32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5\": rpc error: code = NotFound desc = could not find container \"32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5\": container with ID starting with 32fb005dd0237fadc82a37c9ad462e6c55346eaf71d356989c12df5ed6e7c6d5 not found: ID does not exist" Jul 28 01:16:13.732314 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.732296 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b"] Jul 28 01:16:13.735338 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:13.735318 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-60243cce-openshift-default-65b597b8b4-qjm8b"] Jul 28 01:16:14.784723 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:16:14.784693 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d271be-9806-4e8f-9787-10f7140549f6" path="/var/lib/kubelet/pods/13d271be-9806-4e8f-9787-10f7140549f6/volumes" Jul 28 01:22:32.574958 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.574924 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb"] Jul 28 01:22:32.575444 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.575397 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" Jul 28 01:22:32.575444 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.575412 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" Jul 28 01:22:32.575444 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.575428 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" Jul 28 01:22:32.575444 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.575433 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" Jul 28 01:22:32.575579 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.575502 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" Jul 28 01:22:32.578642 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.578622 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.583133 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.583113 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f1343b-openshift-default-dockercfg-64smt\"" Jul 28 01:22:32.592401 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.592381 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb"] Jul 28 01:22:32.680082 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.680049 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-credential-socket\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.680274 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.680091 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-certs\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.680274 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.680180 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-envoy\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.680404 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.680330 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istiod-ca-cert\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.680404 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.680391 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-token\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.680511 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.680421 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-data\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.680511 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.680457 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-podinfo\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.680511 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.680493 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lpp5\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-kube-api-access-4lpp5\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.680664 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.680534 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-socket\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781292 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781268 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-envoy\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781335 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istiod-ca-cert\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781384 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-token\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781408 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-data\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781677 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781435 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-podinfo\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781677 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781469 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lpp5\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-kube-api-access-4lpp5\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781677 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781507 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-socket\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781677 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781545 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-credential-socket\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781677 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781571 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-certs\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781946 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781816 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-data\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.781946 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.781907 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-certs\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.782103 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.782029 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-socket\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.782225 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.782193 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istiod-ca-cert\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.782369 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.782203 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-credential-socket\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.783893 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.783868 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-envoy\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.784104 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.784082 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-podinfo\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.790137 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.790113 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-token\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.790220 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.790154 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lpp5\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-kube-api-access-4lpp5\") pod \"e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:32.888738 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:32.888673 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:33.029844 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:33.029819 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb"] Jul 28 01:22:33.031441 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:22:33.031412 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f1efd6_0faa_47e4_a5a1_e5d7504e1041.slice/crio-63921967107c1506daed43e332353e01b14b452d33685365232c41a8a05f9d44 WatchSource:0}: Error finding container 63921967107c1506daed43e332353e01b14b452d33685365232c41a8a05f9d44: Status 404 returned error can't find the container with id 63921967107c1506daed43e332353e01b14b452d33685365232c41a8a05f9d44 Jul 28 01:22:33.033548 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:33.033518 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:22:33.033616 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:33.033581 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:22:33.033616 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:33.033608 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:22:33.891464 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:33.891424 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" event={"ID":"05f1efd6-0faa-47e4-a5a1-e5d7504e1041","Type":"ContainerStarted","Data":"9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062"} Jul 28 01:22:33.891813 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:33.891480 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" event={"ID":"05f1efd6-0faa-47e4-a5a1-e5d7504e1041","Type":"ContainerStarted","Data":"63921967107c1506daed43e332353e01b14b452d33685365232c41a8a05f9d44"} Jul 28 01:22:33.913469 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:33.913425 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" podStartSLOduration=1.913411406 podStartE2EDuration="1.913411406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:22:33.911929765 +0000 UTC m=+2149.672777777" watchObservedRunningTime="2026-07-28 01:22:33.913411406 +0000 UTC m=+2149.674259419" Jul 28 01:22:34.888949 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:34.888913 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:34.893677 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:34.893651 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:34.896710 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:34.896683 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:34.897438 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:34.897421 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:49.898632 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:49.898602 2550 patch_prober.go:28] interesting pod/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 28 01:22:49.899054 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:49.898660 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 28 01:22:49.988423 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:49.988395 2550 generic.go:356] "Generic (PLEG): container finished" podID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerID="9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062" exitCode=0 Jul 28 01:22:49.988545 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:49.988468 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" event={"ID":"05f1efd6-0faa-47e4-a5a1-e5d7504e1041","Type":"ContainerDied","Data":"9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062"} Jul 28 01:22:49.988848 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:49.988833 2550 scope.go:117] "RemoveContainer" containerID="9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062" Jul 28 01:22:49.989758 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:49.989730 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:22:49.989811 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:49.989798 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:22:49.989850 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:49.989828 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:22:51.002273 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:51.002236 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" event={"ID":"05f1efd6-0faa-47e4-a5a1-e5d7504e1041","Type":"ContainerStarted","Data":"81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636"} Jul 28 01:22:51.889177 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:51.889143 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:51.889355 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:51.889311 2550 patch_prober.go:28] interesting pod/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 28 01:22:51.889400 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:51.889368 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 28 01:22:52.889547 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:52.889507 2550 patch_prober.go:28] interesting pod/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 28 01:22:52.890045 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:52.889572 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 28 01:22:53.889299 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:53.889265 2550 patch_prober.go:28] interesting pod/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 28 01:22:53.889479 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:53.889321 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 28 01:22:54.889145 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:54.889110 2550 patch_prober.go:28] interesting pod/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 28 01:22:54.889553 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:54.889168 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 28 01:22:55.892631 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:55.892600 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:55.893075 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:55.893055 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:22:55.893771 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:22:55.893754 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:23:49.931066 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:49.931026 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n"] Jul 28 01:23:49.931557 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:49.931540 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d271be-9806-4e8f-9787-10f7140549f6" containerName="istio-proxy" Jul 28 01:23:49.934188 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:49.934169 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:49.936733 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:49.936707 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f1343b\"/\"kube-root-ca.crt\"" Jul 28 01:23:49.937532 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:49.937496 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f1343b\"/\"openshift-service-ca.crt\"" Jul 28 01:23:49.937672 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:49.937614 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f1343b\"/\"default-dockercfg-9hrbk\"" Jul 28 01:23:49.937672 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:49.937632 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f1343b\"/\"test-model-f1343b-kserve-self-signed-certs\"" Jul 28 01:23:49.946242 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:49.946215 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n"] Jul 28 01:23:50.043745 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.043721 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-home\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.043843 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.043768 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkzcv\" (UniqueName: \"kubernetes.io/projected/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-kube-api-access-jkzcv\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.043843 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.043823 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-model-cache\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.043928 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.043864 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-dshm\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.043928 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.043893 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tls-certs\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.043928 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.043916 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tmp-dir\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.145308 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.145283 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-home\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.145412 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.145334 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkzcv\" (UniqueName: \"kubernetes.io/projected/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-kube-api-access-jkzcv\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.145468 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.145444 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-model-cache\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.145508 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.145488 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-dshm\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.145560 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.145516 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tls-certs\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.145647 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.145630 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tmp-dir\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.145776 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.145753 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-home\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.145841 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.145762 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-model-cache\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.145923 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.145905 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tmp-dir\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.147846 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.147819 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-dshm\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.147972 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.147954 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tls-certs\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.153850 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.153819 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkzcv\" (UniqueName: \"kubernetes.io/projected/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-kube-api-access-jkzcv\") pod \"test-model-f1343b-kserve-5c7fd64479-k798n\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.246897 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.246826 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:23:50.382104 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:50.382077 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n"] Jul 28 01:23:50.383483 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:23:50.383448 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode00cbf09_a0fc_4feb_a6fe_d7bcd3e63c5e.slice/crio-c44194edadd216074fc5887f26e4bb62d0ed61de643865b0e72c60c839729fa9 WatchSource:0}: Error finding container c44194edadd216074fc5887f26e4bb62d0ed61de643865b0e72c60c839729fa9: Status 404 returned error can't find the container with id c44194edadd216074fc5887f26e4bb62d0ed61de643865b0e72c60c839729fa9 Jul 28 01:23:51.348814 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:51.348768 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" event={"ID":"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e","Type":"ContainerStarted","Data":"02f83efaffdbed0224a311bcd22a9c0f733a2d952767b3ae76e65601e23f9e6e"} Jul 28 01:23:51.349275 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:51.348827 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" event={"ID":"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e","Type":"ContainerStarted","Data":"c44194edadd216074fc5887f26e4bb62d0ed61de643865b0e72c60c839729fa9"} Jul 28 01:23:51.366657 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:23:51.366603 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" podStartSLOduration=2.366585061 podStartE2EDuration="2.366585061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:23:51.364641012 +0000 UTC m=+2227.125489026" watchObservedRunningTime="2026-07-28 01:23:51.366585061 +0000 UTC m=+2227.127433121" Jul 28 01:24:00.247780 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:00.247738 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:24:00.248326 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:00.247808 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:24:00.260501 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:00.260477 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:24:00.412476 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:00.412453 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:24:41.558831 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:41.558792 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n"] Jul 28 01:24:56.577907 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.577853 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" podUID="e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" containerName="main" containerID="cri-o://02f83efaffdbed0224a311bcd22a9c0f733a2d952767b3ae76e65601e23f9e6e" gracePeriod=45 Jul 28 01:24:56.730130 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.730054 2550 generic.go:356] "Generic (PLEG): container finished" podID="e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" containerID="02f83efaffdbed0224a311bcd22a9c0f733a2d952767b3ae76e65601e23f9e6e" exitCode=0 Jul 28 01:24:56.730306 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.730193 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" event={"ID":"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e","Type":"ContainerDied","Data":"02f83efaffdbed0224a311bcd22a9c0f733a2d952767b3ae76e65601e23f9e6e"} Jul 28 01:24:56.834493 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.834442 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:24:56.897402 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897379 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-dshm\") pod \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " Jul 28 01:24:56.897504 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897490 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tls-certs\") pod \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " Jul 28 01:24:56.897558 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897507 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-home\") pod \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " Jul 28 01:24:56.897558 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897539 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkzcv\" (UniqueName: \"kubernetes.io/projected/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-kube-api-access-jkzcv\") pod \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " Jul 28 01:24:56.897670 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897589 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-model-cache\") pod \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " Jul 28 01:24:56.897670 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897603 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tmp-dir\") pod \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\" (UID: \"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e\") " Jul 28 01:24:56.897813 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897772 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-home" (OuterVolumeSpecName: "home") pod "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" (UID: "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:24:56.897894 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897849 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" (UID: "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:24:56.897943 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897893 2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:24:56.897983 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.897931 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" (UID: "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:24:56.899726 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.899698 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-kube-api-access-jkzcv" (OuterVolumeSpecName: "kube-api-access-jkzcv") pod "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" (UID: "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e"). InnerVolumeSpecName "kube-api-access-jkzcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:24:56.899956 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.899936 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" (UID: "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:24:56.900054 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.899940 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-dshm" (OuterVolumeSpecName: "dshm") pod "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" (UID: "e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:24:56.998648 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.998622 2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:24:56.998648 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.998645 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkzcv\" (UniqueName: \"kubernetes.io/projected/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-kube-api-access-jkzcv\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:24:56.998788 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.998661 2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:24:56.998788 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.998676 2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:24:56.998788 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:56.998686 2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:24:57.737248 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:57.737209 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" Jul 28 01:24:57.737655 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:57.737253 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n" event={"ID":"e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e","Type":"ContainerDied","Data":"c44194edadd216074fc5887f26e4bb62d0ed61de643865b0e72c60c839729fa9"} Jul 28 01:24:57.737655 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:57.737302 2550 scope.go:117] "RemoveContainer" containerID="02f83efaffdbed0224a311bcd22a9c0f733a2d952767b3ae76e65601e23f9e6e" Jul 28 01:24:57.760070 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:57.760045 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n"] Jul 28 01:24:57.762348 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:57.762330 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f1343b/test-model-f1343b-kserve-5c7fd64479-k798n"] Jul 28 01:24:58.786355 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:24:58.786315 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" path="/var/lib/kubelet/pods/e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e/volumes" Jul 28 01:25:25.225917 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:25.225864 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb"] Jul 28 01:25:25.226405 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:25.226137 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" containerID="cri-o://81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636" gracePeriod=30 Jul 28 01:25:25.895314 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:25.895272 2550 patch_prober.go:28] interesting pod/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 01:25:25.895479 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:25.895349 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 01:25:30.490700 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.490643 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:25:30.578001 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.577964 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-socket\") pod \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " Jul 28 01:25:30.578178 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.578098 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-credential-socket\") pod \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " Jul 28 01:25:30.578247 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.578212 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lpp5\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-kube-api-access-4lpp5\") pod \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " Jul 28 01:25:30.578305 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.578242 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-token\") pod \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " Jul 28 01:25:30.578305 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.578272 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-podinfo\") pod \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " Jul 28 01:25:30.578305 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.578299 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-data\") pod \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " Jul 28 01:25:30.578458 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.578358 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-envoy\") pod \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " Jul 28 01:25:30.578458 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.578405 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-certs\") pod \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " Jul 28 01:25:30.578458 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.578435 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istiod-ca-cert\") pod \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\" (UID: \"05f1efd6-0faa-47e4-a5a1-e5d7504e1041\") " Jul 28 01:25:30.579152 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.579120 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "05f1efd6-0faa-47e4-a5a1-e5d7504e1041" (UID: "05f1efd6-0faa-47e4-a5a1-e5d7504e1041"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:25:30.579294 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.579180 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "05f1efd6-0faa-47e4-a5a1-e5d7504e1041" (UID: "05f1efd6-0faa-47e4-a5a1-e5d7504e1041"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:25:30.579373 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.579299 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "05f1efd6-0faa-47e4-a5a1-e5d7504e1041" (UID: "05f1efd6-0faa-47e4-a5a1-e5d7504e1041"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:25:30.579441 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.579392 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-data" (OuterVolumeSpecName: "istio-data") pod "05f1efd6-0faa-47e4-a5a1-e5d7504e1041" (UID: "05f1efd6-0faa-47e4-a5a1-e5d7504e1041"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:25:30.579633 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.579606 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "05f1efd6-0faa-47e4-a5a1-e5d7504e1041" (UID: "05f1efd6-0faa-47e4-a5a1-e5d7504e1041"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:25:30.580864 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.580837 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-token" (OuterVolumeSpecName: "istio-token") pod "05f1efd6-0faa-47e4-a5a1-e5d7504e1041" (UID: "05f1efd6-0faa-47e4-a5a1-e5d7504e1041"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:25:30.581168 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.581147 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-kube-api-access-4lpp5" (OuterVolumeSpecName: "kube-api-access-4lpp5") pod "05f1efd6-0faa-47e4-a5a1-e5d7504e1041" (UID: "05f1efd6-0faa-47e4-a5a1-e5d7504e1041"). InnerVolumeSpecName "kube-api-access-4lpp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:25:30.581263 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.581167 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "05f1efd6-0faa-47e4-a5a1-e5d7504e1041" (UID: "05f1efd6-0faa-47e4-a5a1-e5d7504e1041"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:25:30.581319 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.581305 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "05f1efd6-0faa-47e4-a5a1-e5d7504e1041" (UID: "05f1efd6-0faa-47e4-a5a1-e5d7504e1041"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 01:25:30.680049 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.680028 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:25:30.680049 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.680048 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:25:30.680182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.680059 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lpp5\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-kube-api-access-4lpp5\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:25:30.680182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.680067 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:25:30.680182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.680076 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:25:30.680182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.680084 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:25:30.680182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.680091 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:25:30.680182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.680099 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:25:30.680182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.680106 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f1efd6-0faa-47e4-a5a1-e5d7504e1041-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:25:30.935947 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.935919 2550 generic.go:356] "Generic (PLEG): container finished" podID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerID="81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636" exitCode=0 Jul 28 01:25:30.936093 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.935977 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" Jul 28 01:25:30.936093 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.936019 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" event={"ID":"05f1efd6-0faa-47e4-a5a1-e5d7504e1041","Type":"ContainerDied","Data":"81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636"} Jul 28 01:25:30.936093 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.936075 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb" event={"ID":"05f1efd6-0faa-47e4-a5a1-e5d7504e1041","Type":"ContainerDied","Data":"63921967107c1506daed43e332353e01b14b452d33685365232c41a8a05f9d44"} Jul 28 01:25:30.936212 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.936105 2550 scope.go:117] "RemoveContainer" containerID="81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636" Jul 28 01:25:30.945195 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.945177 2550 scope.go:117] "RemoveContainer" containerID="9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062" Jul 28 01:25:30.954324 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.954303 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb"] Jul 28 01:25:30.954917 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.954896 2550 scope.go:117] "RemoveContainer" containerID="81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636" Jul 28 01:25:30.955249 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:25:30.955226 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636\": container with ID starting with 81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636 not found: ID does not exist" containerID="81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636" Jul 28 01:25:30.955321 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.955262 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636"} err="failed to get container status \"81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636\": rpc error: code = NotFound desc = could not find container \"81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636\": container with ID starting with 81e838932f8564764879bdbac1ff6f382086529bc03db9b6e82e2278595fe636 not found: ID does not exist" Jul 28 01:25:30.955321 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.955285 2550 scope.go:117] "RemoveContainer" containerID="9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062" Jul 28 01:25:30.955559 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:25:30.955538 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062\": container with ID starting with 9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062 not found: ID does not exist" containerID="9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062" Jul 28 01:25:30.955636 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.955565 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062"} err="failed to get container status \"9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062\": rpc error: code = NotFound desc = could not find container \"9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062\": container with ID starting with 9b92592b7ff23bda89bb76cd4ca9170afbbf0a9a43d1e8ec46780d57aff76062 not found: ID does not exist" Jul 28 01:25:30.956755 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:30.956735 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f1343b-openshift-default-58bfd5b766-sr2zb"] Jul 28 01:25:32.785396 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:32.785364 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" path="/var/lib/kubelet/pods/05f1efd6-0faa-47e4-a5a1-e5d7504e1041/volumes" Jul 28 01:25:34.707519 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.707488 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb"] Jul 28 01:25:34.708107 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.708087 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" Jul 28 01:25:34.708107 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.708107 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" Jul 28 01:25:34.708266 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.708120 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" containerName="main" Jul 28 01:25:34.708266 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.708129 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" containerName="main" Jul 28 01:25:34.708266 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.708165 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" Jul 28 01:25:34.708266 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.708174 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" Jul 28 01:25:34.708461 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.708281 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" Jul 28 01:25:34.708461 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.708294 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00cbf09-a0fc-4feb-a6fe-d7bcd3e63c5e" containerName="main" Jul 28 01:25:34.708461 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.708307 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f1efd6-0faa-47e4-a5a1-e5d7504e1041" containerName="istio-proxy" Jul 28 01:25:34.713105 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.713085 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.715585 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.715556 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9c8614-openshift-default-dockercfg-pf2pb\"" Jul 28 01:25:34.727260 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.727234 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb"] Jul 28 01:25:34.816661 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.816631 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-token\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.816835 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.816745 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-podinfo\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.816835 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.816794 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-certs\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.816835 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.816819 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-data\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.817028 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.816854 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk98q\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-kube-api-access-rk98q\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.817028 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.816903 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-socket\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.817154 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.817090 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-credential-socket\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.817154 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.817136 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-envoy\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.817264 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.817176 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istiod-ca-cert\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.918601 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.918572 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-token\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.918757 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.918610 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-podinfo\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.918757 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.918629 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-certs\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.918757 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.918646 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-data\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.918757 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.918674 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk98q\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-kube-api-access-rk98q\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.918757 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.918710 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-socket\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.918990 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.918793 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-credential-socket\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.918990 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.918832 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-envoy\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.918990 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.918870 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istiod-ca-cert\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.919166 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.919146 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-data\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.919221 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.919191 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-socket\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.919280 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.919254 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-certs\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.919338 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.919291 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-credential-socket\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.919650 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.919630 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istiod-ca-cert\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.920995 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.920977 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-envoy\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.921321 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.921299 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-podinfo\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.925835 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.925810 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-token\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:34.925979 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:34.925962 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk98q\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-kube-api-access-rk98q\") pod \"e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:35.028516 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:35.028464 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:35.163995 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:35.163967 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb"] Jul 28 01:25:35.164873 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:25:35.164842 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a452e5_7f54_4db8_a4b0_7c20e1abc1f3.slice/crio-f942123484782d6f856f8ff6822d781f6993b439613265adbd7029fe1196f160 WatchSource:0}: Error finding container f942123484782d6f856f8ff6822d781f6993b439613265adbd7029fe1196f160: Status 404 returned error can't find the container with id f942123484782d6f856f8ff6822d781f6993b439613265adbd7029fe1196f160 Jul 28 01:25:35.167067 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:35.167033 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:25:35.167138 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:35.167096 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:25:35.167138 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:35.167124 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:25:35.972272 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:35.972237 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" event={"ID":"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3","Type":"ContainerStarted","Data":"67eed0569d68ebb49dd042243a53a3ae4e9dc6d23e7e87f8e76d63ab86b4713b"} Jul 28 01:25:35.972633 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:35.972295 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" event={"ID":"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3","Type":"ContainerStarted","Data":"f942123484782d6f856f8ff6822d781f6993b439613265adbd7029fe1196f160"} Jul 28 01:25:35.992648 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:35.992597 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" podStartSLOduration=1.992583909 podStartE2EDuration="1.992583909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:35.990357858 +0000 UTC m=+2331.751205883" watchObservedRunningTime="2026-07-28 01:25:35.992583909 +0000 UTC m=+2331.753431921" Jul 28 01:25:36.028613 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:36.028588 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:36.033679 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:36.033659 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:36.977997 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:36.977907 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:36.978954 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:36.978928 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:53.020140 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:53.020100 2550 patch_prober.go:28] interesting pod/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 01:25:53.020553 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:53.020167 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 01:25:54.099216 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:54.099183 2550 generic.go:356] "Generic (PLEG): container finished" podID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerID="67eed0569d68ebb49dd042243a53a3ae4e9dc6d23e7e87f8e76d63ab86b4713b" exitCode=0 Jul 28 01:25:54.099606 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:54.099276 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" event={"ID":"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3","Type":"ContainerDied","Data":"67eed0569d68ebb49dd042243a53a3ae4e9dc6d23e7e87f8e76d63ab86b4713b"} Jul 28 01:25:54.099693 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:54.099673 2550 scope.go:117] "RemoveContainer" containerID="67eed0569d68ebb49dd042243a53a3ae4e9dc6d23e7e87f8e76d63ab86b4713b" Jul 28 01:25:54.100724 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:54.100688 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:25:54.100833 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:54.100760 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:25:54.100833 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:54.100794 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:25:55.106471 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:55.106433 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" event={"ID":"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3","Type":"ContainerStarted","Data":"c8f4d2993b0314c481b52519940d5a35d6de46141f6966269a1edd9d5495b571"} Jul 28 01:25:56.029371 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:56.029338 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:56.029553 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:56.029517 2550 patch_prober.go:28] interesting pod/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 01:25:56.029611 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:56.029563 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 01:25:57.029623 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:57.029588 2550 patch_prober.go:28] interesting pod/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 01:25:57.030066 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:57.029639 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 01:25:58.029836 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:58.029796 2550 patch_prober.go:28] interesting pod/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 28 01:25:58.030339 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:58.029860 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 28 01:25:59.033100 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:59.033067 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:59.033513 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:59.033398 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:25:59.034061 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:25:59.034043 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:26:44.285232 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:44.285189 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb"] Jul 28 01:26:44.285694 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:44.285461 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" containerID="cri-o://c8f4d2993b0314c481b52519940d5a35d6de46141f6966269a1edd9d5495b571" gracePeriod=30 Jul 28 01:26:49.432982 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.432951 2550 generic.go:356] "Generic (PLEG): container finished" podID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerID="c8f4d2993b0314c481b52519940d5a35d6de46141f6966269a1edd9d5495b571" exitCode=0 Jul 28 01:26:49.433403 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.433040 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" event={"ID":"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3","Type":"ContainerDied","Data":"c8f4d2993b0314c481b52519940d5a35d6de46141f6966269a1edd9d5495b571"} Jul 28 01:26:49.433403 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.433101 2550 scope.go:117] "RemoveContainer" containerID="67eed0569d68ebb49dd042243a53a3ae4e9dc6d23e7e87f8e76d63ab86b4713b" Jul 28 01:26:49.535693 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.535669 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:26:49.666102 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666029 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-podinfo\") pod \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " Jul 28 01:26:49.666102 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666099 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-envoy\") pod \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " Jul 28 01:26:49.666257 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666189 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-credential-socket\") pod \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " Jul 28 01:26:49.666257 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666223 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk98q\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-kube-api-access-rk98q\") pod \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " Jul 28 01:26:49.666329 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666261 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istiod-ca-cert\") pod \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " Jul 28 01:26:49.666329 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666315 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-token\") pod \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " Jul 28 01:26:49.666424 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666351 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-socket\") pod \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " Jul 28 01:26:49.666424 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666378 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-data\") pod \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " Jul 28 01:26:49.666424 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666408 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-certs\") pod \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\" (UID: \"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3\") " Jul 28 01:26:49.666581 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666456 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" (UID: "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:26:49.666748 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666668 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" (UID: "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:26:49.666748 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666706 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" (UID: "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:26:49.666961 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666847 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:26:49.666961 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666868 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:26:49.666961 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666882 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:26:49.667170 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.666973 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" (UID: "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:26:49.667277 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.667243 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" (UID: "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:26:49.668299 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.668269 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" (UID: "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 01:26:49.668421 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.668381 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" (UID: "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:26:49.668614 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.668595 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" (UID: "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:26:49.668702 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.668685 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-kube-api-access-rk98q" (OuterVolumeSpecName: "kube-api-access-rk98q") pod "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" (UID: "83a452e5-7f54-4db8-a4b0-7c20e1abc1f3"). InnerVolumeSpecName "kube-api-access-rk98q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:26:49.767287 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.767264 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:26:49.767287 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.767283 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:26:49.767426 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.767294 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk98q\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-kube-api-access-rk98q\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:26:49.767426 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.767302 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:26:49.767426 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.767309 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:26:49.767426 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:49.767317 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:26:50.440256 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:50.440224 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" Jul 28 01:26:50.440698 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:50.440261 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb" event={"ID":"83a452e5-7f54-4db8-a4b0-7c20e1abc1f3","Type":"ContainerDied","Data":"f942123484782d6f856f8ff6822d781f6993b439613265adbd7029fe1196f160"} Jul 28 01:26:50.440698 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:50.440315 2550 scope.go:117] "RemoveContainer" containerID="c8f4d2993b0314c481b52519940d5a35d6de46141f6966269a1edd9d5495b571" Jul 28 01:26:50.462380 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:50.462356 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb"] Jul 28 01:26:50.467164 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:50.467138 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9c8614-openshift-default-696ddccdcb-xm7gb"] Jul 28 01:26:50.788081 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:50.787838 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" path="/var/lib/kubelet/pods/83a452e5-7f54-4db8-a4b0-7c20e1abc1f3/volumes" Jul 28 01:26:52.842054 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.842019 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl"] Jul 28 01:26:52.842518 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.842457 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" Jul 28 01:26:52.842518 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.842468 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" Jul 28 01:26:52.842518 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.842483 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" Jul 28 01:26:52.842518 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.842489 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" Jul 28 01:26:52.842781 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.842552 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" Jul 28 01:26:52.842781 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.842560 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="83a452e5-7f54-4db8-a4b0-7c20e1abc1f3" containerName="istio-proxy" Jul 28 01:26:52.846917 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.846893 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.849552 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.849530 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-3131e3-openshift-default-dockercfg-vqhgz\"" Jul 28 01:26:52.859733 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.859709 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl"] Jul 28 01:26:52.892671 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.892644 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istiod-ca-cert\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.892765 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.892705 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-envoy\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.892810 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.892766 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-data\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.892810 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.892799 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-credential-socket\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.892892 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.892837 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-podinfo\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.892892 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.892887 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8llc\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-kube-api-access-v8llc\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.892972 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.892915 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-socket\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.892972 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.892933 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-token\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.892972 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.892955 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-certs\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994335 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994303 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-envoy\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994515 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994356 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-data\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994515 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994386 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-credential-socket\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994515 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994428 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-podinfo\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994515 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994463 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8llc\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-kube-api-access-v8llc\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994515 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994492 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-socket\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994787 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994516 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-token\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994787 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994551 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-certs\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994787 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994588 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istiod-ca-cert\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.994787 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994762 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-credential-socket\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.995024 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.994817 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-data\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.995093 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.995046 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-certs\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.995176 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.995151 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-socket\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.995331 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.995310 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istiod-ca-cert\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.997133 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.997114 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-envoy\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:52.997373 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:52.997355 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-podinfo\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:53.002227 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.002205 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-token\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:53.002304 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.002243 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8llc\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-kube-api-access-v8llc\") pod \"e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:53.157559 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.157498 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:53.298119 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.298086 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl"] Jul 28 01:26:53.299103 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:26:53.299074 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbceaed1_fe79_4ccb_b86b_4e22f62db39f.slice/crio-be5c9b1ec55972fcb46798a904b81eb0715375c9a180c1250f40127b71b9fc52 WatchSource:0}: Error finding container be5c9b1ec55972fcb46798a904b81eb0715375c9a180c1250f40127b71b9fc52: Status 404 returned error can't find the container with id be5c9b1ec55972fcb46798a904b81eb0715375c9a180c1250f40127b71b9fc52 Jul 28 01:26:53.301627 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.301584 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:26:53.301718 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.301659 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:26:53.301718 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.301696 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:26:53.459263 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.459182 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" event={"ID":"cbceaed1-fe79-4ccb-b86b-4e22f62db39f","Type":"ContainerStarted","Data":"a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177"} Jul 28 01:26:53.459263 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.459235 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" event={"ID":"cbceaed1-fe79-4ccb-b86b-4e22f62db39f","Type":"ContainerStarted","Data":"be5c9b1ec55972fcb46798a904b81eb0715375c9a180c1250f40127b71b9fc52"} Jul 28 01:26:53.478868 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:53.478802 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" podStartSLOduration=1.478781107 podStartE2EDuration="1.478781107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:53.477779639 +0000 UTC m=+2409.238627655" watchObservedRunningTime="2026-07-28 01:26:53.478781107 +0000 UTC m=+2409.239629120" Jul 28 01:26:54.158435 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:54.158405 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:54.163513 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:54.163488 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:54.465067 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:54.464981 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:26:54.465819 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:26:54.465801 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:27:10.507140 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:10.507107 2550 patch_prober.go:28] interesting pod/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 01:27:10.507546 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:10.507180 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 01:27:11.575560 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:11.575529 2550 generic.go:356] "Generic (PLEG): container finished" podID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerID="a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177" exitCode=0 Jul 28 01:27:11.575961 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:11.575603 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" event={"ID":"cbceaed1-fe79-4ccb-b86b-4e22f62db39f","Type":"ContainerDied","Data":"a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177"} Jul 28 01:27:11.576033 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:11.575977 2550 scope.go:117] "RemoveContainer" containerID="a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177" Jul 28 01:27:11.576976 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:11.576945 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:27:11.577108 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:11.577035 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:27:11.577108 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:11.577068 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:27:12.582636 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:12.582603 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" event={"ID":"cbceaed1-fe79-4ccb-b86b-4e22f62db39f","Type":"ContainerStarted","Data":"220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900"} Jul 28 01:27:13.158781 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:13.158749 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:27:13.159175 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:13.159082 2550 patch_prober.go:28] interesting pod/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 28 01:27:13.159379 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:13.159347 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 28 01:27:14.158177 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:14.158141 2550 patch_prober.go:28] interesting pod/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 28 01:27:14.158749 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:14.158718 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 28 01:27:15.158848 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:15.158809 2550 patch_prober.go:28] interesting pod/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 28 01:27:15.159434 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:15.158869 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 28 01:27:16.158328 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:16.158293 2550 patch_prober.go:28] interesting pod/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 28 01:27:16.158599 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:16.158572 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 28 01:27:17.161158 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:17.161130 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:27:17.161564 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:17.161389 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:27:17.161929 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:27:17.161906 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:28:16.819503 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.819461 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw"] Jul 28 01:28:16.823255 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.823233 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:16.826454 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.826225 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"kube-root-ca.crt\"" Jul 28 01:28:16.826454 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.826291 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"default-dockercfg-n8zf4\"" Jul 28 01:28:16.826454 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.826228 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"openshift-service-ca.crt\"" Jul 28 01:28:16.826454 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.826236 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"auth-test-model-3131e3-kserve-self-signed-certs\"" Jul 28 01:28:16.831369 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.831348 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw"] Jul 28 01:28:16.975739 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.975711 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-home\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:16.975849 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.975755 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-model-cache\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:16.975849 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.975779 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2bc6456-7464-439a-a98c-40eb1a34aee8-tls-certs\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:16.975849 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.975830 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-tmp-dir\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:16.975974 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.975855 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrvh\" (UniqueName: \"kubernetes.io/projected/e2bc6456-7464-439a-a98c-40eb1a34aee8-kube-api-access-kcrvh\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:16.975974 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:16.975895 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-dshm\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.077306 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.077229 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-tmp-dir\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.077306 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.077275 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrvh\" (UniqueName: \"kubernetes.io/projected/e2bc6456-7464-439a-a98c-40eb1a34aee8-kube-api-access-kcrvh\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.077306 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.077305 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-dshm\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.077511 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.077358 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-home\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.077511 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.077397 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-model-cache\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.077511 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.077429 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2bc6456-7464-439a-a98c-40eb1a34aee8-tls-certs\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.077713 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.077686 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-tmp-dir\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.077844 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.077795 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-home\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.077844 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.077801 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-model-cache\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.079570 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.079547 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-dshm\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.079934 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.079907 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2bc6456-7464-439a-a98c-40eb1a34aee8-tls-certs\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.085044 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.085022 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrvh\" (UniqueName: \"kubernetes.io/projected/e2bc6456-7464-439a-a98c-40eb1a34aee8-kube-api-access-kcrvh\") pod \"auth-test-model-3131e3-kserve-c5fc5765f-g7hhw\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.134794 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.134772 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:17.272140 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.272116 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw"] Jul 28 01:28:17.272697 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:28:17.272670 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2bc6456_7464_439a_a98c_40eb1a34aee8.slice/crio-5b9332f3b145d0a66017f5b65d06092110eaea157b5aab43c174046cc1f6b839 WatchSource:0}: Error finding container 5b9332f3b145d0a66017f5b65d06092110eaea157b5aab43c174046cc1f6b839: Status 404 returned error can't find the container with id 5b9332f3b145d0a66017f5b65d06092110eaea157b5aab43c174046cc1f6b839 Jul 28 01:28:17.990397 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.990362 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" event={"ID":"e2bc6456-7464-439a-a98c-40eb1a34aee8","Type":"ContainerStarted","Data":"8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464"} Jul 28 01:28:17.990766 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:17.990407 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" event={"ID":"e2bc6456-7464-439a-a98c-40eb1a34aee8","Type":"ContainerStarted","Data":"5b9332f3b145d0a66017f5b65d06092110eaea157b5aab43c174046cc1f6b839"} Jul 28 01:28:18.014045 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:18.011934 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" podStartSLOduration=2.011914298 podStartE2EDuration="2.011914298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:28:18.008700021 +0000 UTC m=+2493.769548034" watchObservedRunningTime="2026-07-28 01:28:18.011914298 +0000 UTC m=+2493.772762312" Jul 28 01:28:27.135461 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:27.135422 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:27.135461 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:27.135468 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:27.148963 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:27.148933 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:28:28.062133 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:28:28.062105 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:32:50.239392 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:32:50.239357 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw"] Jul 28 01:33:05.258343 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.258294 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" podUID="e2bc6456-7464-439a-a98c-40eb1a34aee8" containerName="main" containerID="cri-o://8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464" gracePeriod=45 Jul 28 01:33:05.516743 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.516674 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:33:05.652646 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.652601 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2bc6456-7464-439a-a98c-40eb1a34aee8-tls-certs\") pod \"e2bc6456-7464-439a-a98c-40eb1a34aee8\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " Jul 28 01:33:05.652828 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.652688 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-home\") pod \"e2bc6456-7464-439a-a98c-40eb1a34aee8\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " Jul 28 01:33:05.652828 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.652786 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-model-cache\") pod \"e2bc6456-7464-439a-a98c-40eb1a34aee8\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " Jul 28 01:33:05.652956 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.652842 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-dshm\") pod \"e2bc6456-7464-439a-a98c-40eb1a34aee8\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " Jul 28 01:33:05.652956 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.652918 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcrvh\" (UniqueName: \"kubernetes.io/projected/e2bc6456-7464-439a-a98c-40eb1a34aee8-kube-api-access-kcrvh\") pod \"e2bc6456-7464-439a-a98c-40eb1a34aee8\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " Jul 28 01:33:05.652956 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.652949 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-tmp-dir\") pod \"e2bc6456-7464-439a-a98c-40eb1a34aee8\" (UID: \"e2bc6456-7464-439a-a98c-40eb1a34aee8\") " Jul 28 01:33:05.653160 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.653081 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2bc6456-7464-439a-a98c-40eb1a34aee8" (UID: "e2bc6456-7464-439a-a98c-40eb1a34aee8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:33:05.653344 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.653292 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-home" (OuterVolumeSpecName: "home") pod "e2bc6456-7464-439a-a98c-40eb1a34aee8" (UID: "e2bc6456-7464-439a-a98c-40eb1a34aee8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:33:05.653473 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.653343 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2bc6456-7464-439a-a98c-40eb1a34aee8" (UID: "e2bc6456-7464-439a-a98c-40eb1a34aee8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:33:05.653599 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.653407 2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:33:05.655362 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.655336 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2bc6456-7464-439a-a98c-40eb1a34aee8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2bc6456-7464-439a-a98c-40eb1a34aee8" (UID: "e2bc6456-7464-439a-a98c-40eb1a34aee8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:33:05.655575 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.655539 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bc6456-7464-439a-a98c-40eb1a34aee8-kube-api-access-kcrvh" (OuterVolumeSpecName: "kube-api-access-kcrvh") pod "e2bc6456-7464-439a-a98c-40eb1a34aee8" (UID: "e2bc6456-7464-439a-a98c-40eb1a34aee8"). InnerVolumeSpecName "kube-api-access-kcrvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:33:05.655682 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.655583 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-dshm" (OuterVolumeSpecName: "dshm") pod "e2bc6456-7464-439a-a98c-40eb1a34aee8" (UID: "e2bc6456-7464-439a-a98c-40eb1a34aee8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:33:05.662278 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.662249 2550 generic.go:356] "Generic (PLEG): container finished" podID="e2bc6456-7464-439a-a98c-40eb1a34aee8" containerID="8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464" exitCode=0 Jul 28 01:33:05.662387 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.662294 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" Jul 28 01:33:05.662387 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.662339 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" event={"ID":"e2bc6456-7464-439a-a98c-40eb1a34aee8","Type":"ContainerDied","Data":"8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464"} Jul 28 01:33:05.662504 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.662395 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw" event={"ID":"e2bc6456-7464-439a-a98c-40eb1a34aee8","Type":"ContainerDied","Data":"5b9332f3b145d0a66017f5b65d06092110eaea157b5aab43c174046cc1f6b839"} Jul 28 01:33:05.662504 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.662420 2550 scope.go:117] "RemoveContainer" containerID="8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464" Jul 28 01:33:05.680092 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.680076 2550 scope.go:117] "RemoveContainer" containerID="8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464" Jul 28 01:33:05.680359 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:33:05.680332 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464\": container with ID starting with 8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464 not found: ID does not exist" containerID="8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464" Jul 28 01:33:05.680410 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.680369 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464"} err="failed to get container status \"8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464\": rpc error: code = NotFound desc = could not find container \"8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464\": container with ID starting with 8da62714001a9ac3cfab3e35de8e2a2989dd1afa3e7bc922fe620f9fe9642464 not found: ID does not exist" Jul 28 01:33:05.688748 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.688728 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw"] Jul 28 01:33:05.692559 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.692538 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/auth-test-model-3131e3-kserve-c5fc5765f-g7hhw"] Jul 28 01:33:05.754042 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.753999 2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2bc6456-7464-439a-a98c-40eb1a34aee8-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:33:05.754042 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.754041 2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:33:05.754192 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.754049 2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:33:05.754192 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.754058 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcrvh\" (UniqueName: \"kubernetes.io/projected/e2bc6456-7464-439a-a98c-40eb1a34aee8-kube-api-access-kcrvh\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:33:05.754192 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:05.754069 2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2bc6456-7464-439a-a98c-40eb1a34aee8-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:33:06.783907 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:33:06.783827 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2bc6456-7464-439a-a98c-40eb1a34aee8" path="/var/lib/kubelet/pods/e2bc6456-7464-439a-a98c-40eb1a34aee8/volumes" Jul 28 01:34:56.605488 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.605410 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc"] Jul 28 01:34:56.605937 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.605801 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2bc6456-7464-439a-a98c-40eb1a34aee8" containerName="main" Jul 28 01:34:56.605937 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.605813 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2bc6456-7464-439a-a98c-40eb1a34aee8" containerName="main" Jul 28 01:34:56.605937 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.605886 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2bc6456-7464-439a-a98c-40eb1a34aee8" containerName="main" Jul 28 01:34:56.609001 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.608981 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.613402 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.613382 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-14a1d0\"/\"kube-root-ca.crt\"" Jul 28 01:34:56.613558 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.613545 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-14a1d0\"/\"openshift-service-ca.crt\"" Jul 28 01:34:56.614521 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.614499 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-14a1d0\"/\"default-dockercfg-kvlf8\"" Jul 28 01:34:56.614607 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.614591 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-14a1d0\"/\"rate-test-model-14a1d0-kserve-self-signed-certs\"" Jul 28 01:34:56.620906 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.620884 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc"] Jul 28 01:34:56.667943 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.667920 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-tmp-dir\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.668057 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.667962 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-dshm\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.668057 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.667991 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdfp7\" (UniqueName: \"kubernetes.io/projected/90cedbba-7898-4c13-9ca3-8127d3067854-kube-api-access-fdfp7\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.668135 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.668071 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cedbba-7898-4c13-9ca3-8127d3067854-tls-certs\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.668135 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.668106 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-model-cache\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.668135 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.668123 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-home\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.769024 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.768984 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-tmp-dir\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.769134 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.769042 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-dshm\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.769134 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.769069 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdfp7\" (UniqueName: \"kubernetes.io/projected/90cedbba-7898-4c13-9ca3-8127d3067854-kube-api-access-fdfp7\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.769134 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.769093 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cedbba-7898-4c13-9ca3-8127d3067854-tls-certs\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.769134 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.769112 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-model-cache\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.769134 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.769126 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-home\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.769446 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.769426 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-tmp-dir\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.769513 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.769466 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-model-cache\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.769576 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.769549 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-home\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.771207 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.771183 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-dshm\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.771535 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.771519 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cedbba-7898-4c13-9ca3-8127d3067854-tls-certs\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.776602 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.776581 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdfp7\" (UniqueName: \"kubernetes.io/projected/90cedbba-7898-4c13-9ca3-8127d3067854-kube-api-access-fdfp7\") pod \"rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:56.918775 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:56.918680 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:34:57.052830 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:57.052801 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc"] Jul 28 01:34:57.054688 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:34:57.054662 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90cedbba_7898_4c13_9ca3_8127d3067854.slice/crio-2a826739dd98bfff16d76a88727b7a8d422759901846032b52d282eda549beb7 WatchSource:0}: Error finding container 2a826739dd98bfff16d76a88727b7a8d422759901846032b52d282eda549beb7: Status 404 returned error can't find the container with id 2a826739dd98bfff16d76a88727b7a8d422759901846032b52d282eda549beb7 Jul 28 01:34:57.303693 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:57.303662 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" event={"ID":"90cedbba-7898-4c13-9ca3-8127d3067854","Type":"ContainerStarted","Data":"1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be"} Jul 28 01:34:57.303843 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:57.303703 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" event={"ID":"90cedbba-7898-4c13-9ca3-8127d3067854","Type":"ContainerStarted","Data":"2a826739dd98bfff16d76a88727b7a8d422759901846032b52d282eda549beb7"} Jul 28 01:34:57.320490 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:34:57.320445 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" podStartSLOduration=1.3204298780000001 podStartE2EDuration="1.320429878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:57.31990505 +0000 UTC m=+2893.080753063" watchObservedRunningTime="2026-07-28 01:34:57.320429878 +0000 UTC m=+2893.081277892" Jul 28 01:35:06.919318 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:35:06.919278 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:35:06.919954 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:35:06.919332 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:35:06.931684 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:35:06.931659 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:35:07.382505 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:35:07.382476 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:36:36.412113 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:36.412081 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc"] Jul 28 01:36:51.431291 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.431242 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" podUID="90cedbba-7898-4c13-9ca3-8127d3067854" containerName="main" containerID="cri-o://1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be" gracePeriod=45 Jul 28 01:36:51.684077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.684025 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:36:51.712163 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712136 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cedbba-7898-4c13-9ca3-8127d3067854-tls-certs\") pod \"90cedbba-7898-4c13-9ca3-8127d3067854\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " Jul 28 01:36:51.712278 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712203 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-home\") pod \"90cedbba-7898-4c13-9ca3-8127d3067854\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " Jul 28 01:36:51.712378 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712361 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdfp7\" (UniqueName: \"kubernetes.io/projected/90cedbba-7898-4c13-9ca3-8127d3067854-kube-api-access-fdfp7\") pod \"90cedbba-7898-4c13-9ca3-8127d3067854\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " Jul 28 01:36:51.712444 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712398 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-dshm\") pod \"90cedbba-7898-4c13-9ca3-8127d3067854\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " Jul 28 01:36:51.712444 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712431 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-home" (OuterVolumeSpecName: "home") pod "90cedbba-7898-4c13-9ca3-8127d3067854" (UID: "90cedbba-7898-4c13-9ca3-8127d3067854"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:36:51.712553 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712463 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-model-cache\") pod \"90cedbba-7898-4c13-9ca3-8127d3067854\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " Jul 28 01:36:51.712553 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712501 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-tmp-dir\") pod \"90cedbba-7898-4c13-9ca3-8127d3067854\" (UID: \"90cedbba-7898-4c13-9ca3-8127d3067854\") " Jul 28 01:36:51.712795 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712746 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-model-cache" (OuterVolumeSpecName: "model-cache") pod "90cedbba-7898-4c13-9ca3-8127d3067854" (UID: "90cedbba-7898-4c13-9ca3-8127d3067854"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:36:51.712901 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712805 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90cedbba-7898-4c13-9ca3-8127d3067854" (UID: "90cedbba-7898-4c13-9ca3-8127d3067854"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:36:51.712946 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712903 2550 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-model-cache\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:36:51.713001 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712944 2550 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-tmp-dir\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:36:51.713001 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.712958 2550 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-home\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:36:51.714423 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.714404 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90cedbba-7898-4c13-9ca3-8127d3067854-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90cedbba-7898-4c13-9ca3-8127d3067854" (UID: "90cedbba-7898-4c13-9ca3-8127d3067854"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 01:36:51.714531 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.714514 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90cedbba-7898-4c13-9ca3-8127d3067854-kube-api-access-fdfp7" (OuterVolumeSpecName: "kube-api-access-fdfp7") pod "90cedbba-7898-4c13-9ca3-8127d3067854" (UID: "90cedbba-7898-4c13-9ca3-8127d3067854"). InnerVolumeSpecName "kube-api-access-fdfp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:36:51.714745 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.714726 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-dshm" (OuterVolumeSpecName: "dshm") pod "90cedbba-7898-4c13-9ca3-8127d3067854" (UID: "90cedbba-7898-4c13-9ca3-8127d3067854"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:36:51.814346 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.814311 2550 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90cedbba-7898-4c13-9ca3-8127d3067854-tls-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:36:51.814346 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.814343 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdfp7\" (UniqueName: \"kubernetes.io/projected/90cedbba-7898-4c13-9ca3-8127d3067854-kube-api-access-fdfp7\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:36:51.814346 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.814353 2550 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90cedbba-7898-4c13-9ca3-8127d3067854-dshm\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:36:51.999469 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.999398 2550 generic.go:356] "Generic (PLEG): container finished" podID="90cedbba-7898-4c13-9ca3-8127d3067854" containerID="1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be" exitCode=0 Jul 28 01:36:51.999469 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.999437 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" Jul 28 01:36:51.999610 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.999491 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" event={"ID":"90cedbba-7898-4c13-9ca3-8127d3067854","Type":"ContainerDied","Data":"1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be"} Jul 28 01:36:51.999610 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.999529 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc" event={"ID":"90cedbba-7898-4c13-9ca3-8127d3067854","Type":"ContainerDied","Data":"2a826739dd98bfff16d76a88727b7a8d422759901846032b52d282eda549beb7"} Jul 28 01:36:51.999610 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:51.999548 2550 scope.go:117] "RemoveContainer" containerID="1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be" Jul 28 01:36:52.010638 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:52.010498 2550 scope.go:117] "RemoveContainer" containerID="1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be" Jul 28 01:36:52.010798 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:36:52.010774 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be\": container with ID starting with 1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be not found: ID does not exist" containerID="1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be" Jul 28 01:36:52.010911 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:52.010809 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be"} err="failed to get container status \"1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be\": rpc error: code = NotFound desc = could not find container \"1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be\": container with ID starting with 1746fe9e2dcabc056d92a3697a30da9a0d2c1e62b62c4f5a1cb389219ed117be not found: ID does not exist" Jul 28 01:36:52.023997 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:52.023972 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc"] Jul 28 01:36:52.027783 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:52.027759 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/rate-test-model-14a1d0-kserve-56bfbcd54c-hv9sc"] Jul 28 01:36:52.784558 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:36:52.784528 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90cedbba-7898-4c13-9ca3-8127d3067854" path="/var/lib/kubelet/pods/90cedbba-7898-4c13-9ca3-8127d3067854/volumes" Jul 28 01:37:15.496335 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.496305 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w"] Jul 28 01:37:15.496941 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.496921 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90cedbba-7898-4c13-9ca3-8127d3067854" containerName="main" Jul 28 01:37:15.497060 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.496945 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="90cedbba-7898-4c13-9ca3-8127d3067854" containerName="main" Jul 28 01:37:15.497128 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.497111 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="90cedbba-7898-4c13-9ca3-8127d3067854" containerName="main" Jul 28 01:37:15.500294 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.500273 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.502473 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.502450 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1ce59a-openshift-default-dockercfg-djgt4\"" Jul 28 01:37:15.510869 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.510844 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w"] Jul 28 01:37:15.617077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.614322 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istiod-ca-cert\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.617077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.614393 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-token\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.617077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.614428 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-data\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.617077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.614478 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-credential-socket\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.617077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.614508 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-socket\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.617077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.614540 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-envoy\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.617077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.614563 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-podinfo\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.617077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.614606 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-certs\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.617077 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.614629 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zssq\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-kube-api-access-7zssq\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.715711 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.715686 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-credential-socket\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.715854 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.715716 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-socket\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.715854 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.715736 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-envoy\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.715854 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.715752 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-podinfo\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.715854 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.715787 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-certs\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.716090 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.715946 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zssq\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-kube-api-access-7zssq\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.716090 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.716063 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istiod-ca-cert\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.716198 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.716114 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-token\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.716198 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.716149 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-data\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.716198 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.716167 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-socket\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.716384 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.716356 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-credential-socket\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.716505 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.716411 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-certs\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.716505 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.716490 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-data\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.716923 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.716900 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istiod-ca-cert\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.718239 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.718218 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-envoy\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.718853 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.718833 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-podinfo\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.723268 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.723244 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-token\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.723638 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.723617 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zssq\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-kube-api-access-7zssq\") pod \"e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.811143 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.811121 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:15.947414 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.947383 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w"] Jul 28 01:37:15.949067 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:37:15.949044 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc667ec74_461c_4ebd_b04c_2f79bf7cbee2.slice/crio-0332f1e22a3a306ed59757c8a2ba381635772ab1c8bdf4b93b23b5b63f38ac3f WatchSource:0}: Error finding container 0332f1e22a3a306ed59757c8a2ba381635772ab1c8bdf4b93b23b5b63f38ac3f: Status 404 returned error can't find the container with id 0332f1e22a3a306ed59757c8a2ba381635772ab1c8bdf4b93b23b5b63f38ac3f Jul 28 01:37:15.951712 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.951680 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:37:15.951798 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.951742 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:37:15.951798 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:15.951768 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:37:16.141067 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:16.140935 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" event={"ID":"c667ec74-461c-4ebd-b04c-2f79bf7cbee2","Type":"ContainerStarted","Data":"67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1"} Jul 28 01:37:16.141067 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:16.141038 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" event={"ID":"c667ec74-461c-4ebd-b04c-2f79bf7cbee2","Type":"ContainerStarted","Data":"0332f1e22a3a306ed59757c8a2ba381635772ab1c8bdf4b93b23b5b63f38ac3f"} Jul 28 01:37:16.160378 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:16.160321 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" podStartSLOduration=1.160302389 podStartE2EDuration="1.160302389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:16.159341525 +0000 UTC m=+3031.920189537" watchObservedRunningTime="2026-07-28 01:37:16.160302389 +0000 UTC m=+3031.921150402" Jul 28 01:37:16.812088 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:16.812058 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:17.816580 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:17.816544 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:18.153909 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:18.153836 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:18.154787 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:18.154764 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:31.233145 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:31.233108 2550 generic.go:356] "Generic (PLEG): container finished" podID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerID="67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1" exitCode=0 Jul 28 01:37:31.233553 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:31.233168 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" event={"ID":"c667ec74-461c-4ebd-b04c-2f79bf7cbee2","Type":"ContainerDied","Data":"67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1"} Jul 28 01:37:31.233553 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:31.233546 2550 scope.go:117] "RemoveContainer" containerID="67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1" Jul 28 01:37:31.234671 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:31.234641 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:37:31.234733 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:31.234711 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:37:31.234770 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:31.234744 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:37:32.240945 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:32.240907 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" event={"ID":"c667ec74-461c-4ebd-b04c-2f79bf7cbee2","Type":"ContainerStarted","Data":"b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1"} Jul 28 01:37:32.811638 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:32.811604 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:32.812865 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:32.812839 2550 patch_prober.go:28] interesting pod/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 28 01:37:32.812980 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:32.812890 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" podUID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 28 01:37:33.153797 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:33.153710 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:33.811880 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:33.811844 2550 patch_prober.go:28] interesting pod/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 28 01:37:33.812285 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:33.811897 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" podUID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 28 01:37:34.812241 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:34.812197 2550 patch_prober.go:28] interesting pod/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 28 01:37:34.812719 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:34.812247 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" podUID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 28 01:37:35.812024 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:35.811964 2550 patch_prober.go:28] interesting pod/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 28 01:37:35.812233 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:35.812042 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" podUID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 28 01:37:36.815518 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:36.815445 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:37:36.816291 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:37:36.816271 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:38:37.928610 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:37.928581 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq"] Jul 28 01:38:37.932190 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:37.932173 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:37.935100 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:37.935078 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-5ecc77-openshift-default-dockercfg-pdlwg\"" Jul 28 01:38:37.945521 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:37.945500 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq"] Jul 28 01:38:38.014272 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.014248 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-data\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.014391 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.014285 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66382a3e-6018-4bb0-bcd4-233f7986b493-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.014391 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.014315 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-podinfo\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.014472 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.014399 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-credential-socket\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.014472 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.014435 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-certs\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.014541 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.014470 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sncb\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-kube-api-access-7sncb\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.014541 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.014515 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-socket\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.014613 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.014592 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-envoy\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.014670 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.014641 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-token\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115188 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115149 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-socket\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115188 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115195 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-envoy\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115228 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-token\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115246 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-data\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115271 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66382a3e-6018-4bb0-bcd4-233f7986b493-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115307 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-podinfo\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115341 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-credential-socket\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115382 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-certs\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115430 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115424 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sncb\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-kube-api-access-7sncb\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115861 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115588 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-socket\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115861 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115743 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-data\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115989 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115939 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-credential-socket\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.115989 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.115957 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-certs\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.116159 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.116133 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66382a3e-6018-4bb0-bcd4-233f7986b493-istiod-ca-cert\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.118193 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.118164 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-envoy\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.118339 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.118321 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-podinfo\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.123810 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.123781 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sncb\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-kube-api-access-7sncb\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.123969 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.123945 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-token\") pod \"e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.243239 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.243181 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:38.392847 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.392819 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq"] Jul 28 01:38:38.394616 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:38:38.394587 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66382a3e_6018_4bb0_bcd4_233f7986b493.slice/crio-20fb6b1e8d743850d53cbeb7f356c1daec0ad5a845a33f6ebe1c665636a48552 WatchSource:0}: Error finding container 20fb6b1e8d743850d53cbeb7f356c1daec0ad5a845a33f6ebe1c665636a48552: Status 404 returned error can't find the container with id 20fb6b1e8d743850d53cbeb7f356c1daec0ad5a845a33f6ebe1c665636a48552 Jul 28 01:38:38.397063 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.396986 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:38:38.397136 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.397103 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:38:38.397173 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.397156 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:38:38.650603 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.650560 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" event={"ID":"66382a3e-6018-4bb0-bcd4-233f7986b493","Type":"ContainerStarted","Data":"3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf"} Jul 28 01:38:38.650603 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.650609 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" event={"ID":"66382a3e-6018-4bb0-bcd4-233f7986b493","Type":"ContainerStarted","Data":"20fb6b1e8d743850d53cbeb7f356c1daec0ad5a845a33f6ebe1c665636a48552"} Jul 28 01:38:38.669538 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:38.669479 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" podStartSLOduration=1.669452918 podStartE2EDuration="1.669452918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:38.669254151 +0000 UTC m=+3114.430102171" watchObservedRunningTime="2026-07-28 01:38:38.669452918 +0000 UTC m=+3114.430300932" Jul 28 01:38:39.243319 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:39.243287 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:39.248125 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:39.248103 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:39.656873 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:39.656839 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:39.657714 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:39.657693 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:38:44.343254 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:44.343222 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hh65b"] Jul 28 01:38:44.347604 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:44.347584 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:44.357509 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:44.357482 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hh65b"] Jul 28 01:38:44.369563 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:44.369540 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlwk6\" (UniqueName: \"kubernetes.io/projected/d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c-kube-api-access-nlwk6\") pod \"kuadrant-operator-catalog-hh65b\" (UID: \"d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c\") " pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:44.470658 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:44.470632 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlwk6\" (UniqueName: \"kubernetes.io/projected/d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c-kube-api-access-nlwk6\") pod \"kuadrant-operator-catalog-hh65b\" (UID: \"d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c\") " pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:44.480900 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:44.480878 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlwk6\" (UniqueName: \"kubernetes.io/projected/d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c-kube-api-access-nlwk6\") pod \"kuadrant-operator-catalog-hh65b\" (UID: \"d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c\") " pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:44.658514 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:44.658444 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:44.790466 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:44.790439 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hh65b"] Jul 28 01:38:44.791731 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:38:44.791706 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9af9b7d_f18d_4d1b_a5e2_04cdf01b606c.slice/crio-2bf15d074b1242e8506c0e5dc6bb0fb365bca3c5b55630ae5cd14693876caecc WatchSource:0}: Error finding container 2bf15d074b1242e8506c0e5dc6bb0fb365bca3c5b55630ae5cd14693876caecc: Status 404 returned error can't find the container with id 2bf15d074b1242e8506c0e5dc6bb0fb365bca3c5b55630ae5cd14693876caecc Jul 28 01:38:44.793179 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:44.793156 2550 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 01:38:45.699209 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:45.699175 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" event={"ID":"d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c","Type":"ContainerStarted","Data":"37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d"} Jul 28 01:38:45.699574 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:45.699231 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" event={"ID":"d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c","Type":"ContainerStarted","Data":"2bf15d074b1242e8506c0e5dc6bb0fb365bca3c5b55630ae5cd14693876caecc"} Jul 28 01:38:45.716277 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:45.716228 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" podStartSLOduration=1.300526976 podStartE2EDuration="1.716214328s" totalImagesPullingTime="415.687352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:45.714987498 +0000 UTC m=+3121.475835515" watchObservedRunningTime="2026-07-28 01:38:45.716214328 +0000 UTC m=+3121.477062345" Jul 28 01:38:54.659318 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:54.659283 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:54.659805 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:54.659350 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:54.689135 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:54.689102 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:54.788432 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:54.788406 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:54.921097 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:54.921021 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hh65b"] Jul 28 01:38:56.766898 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:56.766854 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" podUID="d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c" containerName="registry-server" containerID="cri-o://37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d" gracePeriod=2 Jul 28 01:38:57.015535 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.015510 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:57.098125 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.098093 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlwk6\" (UniqueName: \"kubernetes.io/projected/d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c-kube-api-access-nlwk6\") pod \"d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c\" (UID: \"d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c\") " Jul 28 01:38:57.100136 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.100111 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c-kube-api-access-nlwk6" (OuterVolumeSpecName: "kube-api-access-nlwk6") pod "d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c" (UID: "d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c"). InnerVolumeSpecName "kube-api-access-nlwk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:38:57.199241 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.199213 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlwk6\" (UniqueName: \"kubernetes.io/projected/d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c-kube-api-access-nlwk6\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:38:57.774287 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.774253 2550 generic.go:356] "Generic (PLEG): container finished" podID="d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c" containerID="37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d" exitCode=0 Jul 28 01:38:57.775612 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.774298 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" Jul 28 01:38:57.775612 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.774336 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" event={"ID":"d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c","Type":"ContainerDied","Data":"37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d"} Jul 28 01:38:57.775612 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.774392 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-hh65b" event={"ID":"d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c","Type":"ContainerDied","Data":"2bf15d074b1242e8506c0e5dc6bb0fb365bca3c5b55630ae5cd14693876caecc"} Jul 28 01:38:57.775612 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.774415 2550 scope.go:117] "RemoveContainer" containerID="37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d" Jul 28 01:38:57.785071 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.785053 2550 scope.go:117] "RemoveContainer" containerID="37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d" Jul 28 01:38:57.785350 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:38:57.785325 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d\": container with ID starting with 37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d not found: ID does not exist" containerID="37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d" Jul 28 01:38:57.785438 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.785358 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d"} err="failed to get container status \"37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d\": rpc error: code = NotFound desc = could not find container \"37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d\": container with ID starting with 37ec93d7dc70562153366faadbc7580aa63c7193ac5c85e2b44f428524339e9d not found: ID does not exist" Jul 28 01:38:57.802445 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.798414 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hh65b"] Jul 28 01:38:57.805731 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:57.805709 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-hh65b"] Jul 28 01:38:58.788825 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:58.788784 2550 generic.go:356] "Generic (PLEG): container finished" podID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerID="3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf" exitCode=0 Jul 28 01:38:58.798669 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:58.798631 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c" path="/var/lib/kubelet/pods/d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c/volumes" Jul 28 01:38:58.799016 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:58.798982 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" event={"ID":"66382a3e-6018-4bb0-bcd4-233f7986b493","Type":"ContainerDied","Data":"3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf"} Jul 28 01:38:58.799760 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:58.799737 2550 scope.go:117] "RemoveContainer" containerID="3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf" Jul 28 01:38:58.800836 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:58.800806 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:38:58.800896 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:58.800872 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:38:58.800939 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:58.800912 2550 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 01:38:59.797896 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:38:59.797861 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" event={"ID":"66382a3e-6018-4bb0-bcd4-233f7986b493","Type":"ContainerStarted","Data":"4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd"} Jul 28 01:39:00.243421 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:00.243326 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:39:00.243574 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:00.243487 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 28 01:39:00.243574 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:00.243532 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 28 01:39:01.244284 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:01.244249 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 28 01:39:01.244665 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:01.244312 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 28 01:39:02.243457 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:02.243422 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 28 01:39:02.243623 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:02.243478 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 28 01:39:03.244475 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:03.244438 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 28 01:39:03.244850 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:03.244496 2550 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 28 01:39:04.247181 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:04.247150 2550 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:39:04.247607 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:04.247437 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:39:04.248214 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:04.248190 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:39:18.211672 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:18.211586 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq"] Jul 28 01:39:18.212235 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:18.211872 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerName="istio-proxy" containerID="cri-o://4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd" gracePeriod=30 Jul 28 01:39:19.249439 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:19.249404 2550 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 01:39:19.249818 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:19.249467 2550 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 01:39:23.470304 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.470248 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:39:23.541844 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.541819 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-socket\") pod \"66382a3e-6018-4bb0-bcd4-233f7986b493\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " Jul 28 01:39:23.541977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.541894 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-certs\") pod \"66382a3e-6018-4bb0-bcd4-233f7986b493\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " Jul 28 01:39:23.541977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.541927 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-data\") pod \"66382a3e-6018-4bb0-bcd4-233f7986b493\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " Jul 28 01:39:23.541977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.541943 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-token\") pod \"66382a3e-6018-4bb0-bcd4-233f7986b493\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " Jul 28 01:39:23.541977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.541958 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-envoy\") pod \"66382a3e-6018-4bb0-bcd4-233f7986b493\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " Jul 28 01:39:23.541977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.541975 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sncb\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-kube-api-access-7sncb\") pod \"66382a3e-6018-4bb0-bcd4-233f7986b493\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " Jul 28 01:39:23.542273 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.541995 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66382a3e-6018-4bb0-bcd4-233f7986b493-istiod-ca-cert\") pod \"66382a3e-6018-4bb0-bcd4-233f7986b493\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " Jul 28 01:39:23.542273 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542114 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "66382a3e-6018-4bb0-bcd4-233f7986b493" (UID: "66382a3e-6018-4bb0-bcd4-233f7986b493"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:39:23.542273 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542139 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-credential-socket\") pod \"66382a3e-6018-4bb0-bcd4-233f7986b493\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " Jul 28 01:39:23.542273 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542178 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-podinfo\") pod \"66382a3e-6018-4bb0-bcd4-233f7986b493\" (UID: \"66382a3e-6018-4bb0-bcd4-233f7986b493\") " Jul 28 01:39:23.542273 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542238 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "66382a3e-6018-4bb0-bcd4-233f7986b493" (UID: "66382a3e-6018-4bb0-bcd4-233f7986b493"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:39:23.542530 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542416 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "66382a3e-6018-4bb0-bcd4-233f7986b493" (UID: "66382a3e-6018-4bb0-bcd4-233f7986b493"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:39:23.542530 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542432 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66382a3e-6018-4bb0-bcd4-233f7986b493-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "66382a3e-6018-4bb0-bcd4-233f7986b493" (UID: "66382a3e-6018-4bb0-bcd4-233f7986b493"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:39:23.542635 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542546 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:39:23.542635 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542566 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:39:23.542635 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542579 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66382a3e-6018-4bb0-bcd4-233f7986b493-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:39:23.542635 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542591 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:39:23.542937 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.542915 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-data" (OuterVolumeSpecName: "istio-data") pod "66382a3e-6018-4bb0-bcd4-233f7986b493" (UID: "66382a3e-6018-4bb0-bcd4-233f7986b493"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:39:23.544280 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.544254 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "66382a3e-6018-4bb0-bcd4-233f7986b493" (UID: "66382a3e-6018-4bb0-bcd4-233f7986b493"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 01:39:23.544393 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.544260 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-token" (OuterVolumeSpecName: "istio-token") pod "66382a3e-6018-4bb0-bcd4-233f7986b493" (UID: "66382a3e-6018-4bb0-bcd4-233f7986b493"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:39:23.544531 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.544513 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-kube-api-access-7sncb" (OuterVolumeSpecName: "kube-api-access-7sncb") pod "66382a3e-6018-4bb0-bcd4-233f7986b493" (UID: "66382a3e-6018-4bb0-bcd4-233f7986b493"). InnerVolumeSpecName "kube-api-access-7sncb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:39:23.544595 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.544581 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "66382a3e-6018-4bb0-bcd4-233f7986b493" (UID: "66382a3e-6018-4bb0-bcd4-233f7986b493"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:39:23.643215 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.643194 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:39:23.643215 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.643214 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:39:23.643332 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.643222 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:39:23.643332 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.643230 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sncb\" (UniqueName: \"kubernetes.io/projected/66382a3e-6018-4bb0-bcd4-233f7986b493-kube-api-access-7sncb\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:39:23.643332 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.643239 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66382a3e-6018-4bb0-bcd4-233f7986b493-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:39:23.933711 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.933680 2550 generic.go:356] "Generic (PLEG): container finished" podID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerID="4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd" exitCode=0 Jul 28 01:39:23.933822 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.933799 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" Jul 28 01:39:23.933822 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.933800 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" event={"ID":"66382a3e-6018-4bb0-bcd4-233f7986b493","Type":"ContainerDied","Data":"4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd"} Jul 28 01:39:23.933912 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.933858 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq" event={"ID":"66382a3e-6018-4bb0-bcd4-233f7986b493","Type":"ContainerDied","Data":"20fb6b1e8d743850d53cbeb7f356c1daec0ad5a845a33f6ebe1c665636a48552"} Jul 28 01:39:23.933965 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.933934 2550 scope.go:117] "RemoveContainer" containerID="4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd" Jul 28 01:39:23.945927 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.945906 2550 scope.go:117] "RemoveContainer" containerID="3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf" Jul 28 01:39:23.954506 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.954490 2550 scope.go:117] "RemoveContainer" containerID="4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd" Jul 28 01:39:23.954769 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:39:23.954751 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd\": container with ID starting with 4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd not found: ID does not exist" containerID="4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd" Jul 28 01:39:23.954817 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.954777 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd"} err="failed to get container status \"4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd\": rpc error: code = NotFound desc = could not find container \"4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd\": container with ID starting with 4f30f4e28205135a5c8134e3862419d31d68c8a2b9bb7b9063a2f9208aed16cd not found: ID does not exist" Jul 28 01:39:23.954817 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.954796 2550 scope.go:117] "RemoveContainer" containerID="3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf" Jul 28 01:39:23.955040 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:39:23.955001 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf\": container with ID starting with 3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf not found: ID does not exist" containerID="3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf" Jul 28 01:39:23.955152 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.955041 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf"} err="failed to get container status \"3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf\": rpc error: code = NotFound desc = could not find container \"3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf\": container with ID starting with 3b6241e4c18c910bfdd181025f6599ecc9ffdd0d31beeedc6bc09a6653f54fdf not found: ID does not exist" Jul 28 01:39:23.964690 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.964666 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq"] Jul 28 01:39:23.968678 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:23.968659 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-5ecc77-openshift-default-756664f87b-jcfpq"] Jul 28 01:39:24.784437 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:39:24.784401 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" path="/var/lib/kubelet/pods/66382a3e-6018-4bb0-bcd4-233f7986b493/volumes" Jul 28 01:40:52.945134 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:52.945050 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w"] Jul 28 01:40:52.945719 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:52.945273 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" podUID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerName="istio-proxy" containerID="cri-o://b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1" gracePeriod=30 Jul 28 01:40:58.202026 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.201983 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:40:58.343065 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343038 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-data\") pod \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " Jul 28 01:40:58.343192 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343069 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-podinfo\") pod \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " Jul 28 01:40:58.343192 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343090 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-socket\") pod \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " Jul 28 01:40:58.343192 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343152 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zssq\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-kube-api-access-7zssq\") pod \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " Jul 28 01:40:58.343192 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343168 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-envoy\") pod \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " Jul 28 01:40:58.343427 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343236 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-certs\") pod \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " Jul 28 01:40:58.343427 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343272 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istiod-ca-cert\") pod \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " Jul 28 01:40:58.343427 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343326 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-credential-socket\") pod \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " Jul 28 01:40:58.343427 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343350 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-token\") pod \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\" (UID: \"c667ec74-461c-4ebd-b04c-2f79bf7cbee2\") " Jul 28 01:40:58.343643 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343612 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c667ec74-461c-4ebd-b04c-2f79bf7cbee2" (UID: "c667ec74-461c-4ebd-b04c-2f79bf7cbee2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:40:58.343698 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343678 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c667ec74-461c-4ebd-b04c-2f79bf7cbee2" (UID: "c667ec74-461c-4ebd-b04c-2f79bf7cbee2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:40:58.343961 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343830 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c667ec74-461c-4ebd-b04c-2f79bf7cbee2" (UID: "c667ec74-461c-4ebd-b04c-2f79bf7cbee2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:40:58.343961 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.343908 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c667ec74-461c-4ebd-b04c-2f79bf7cbee2" (UID: "c667ec74-461c-4ebd-b04c-2f79bf7cbee2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:40:58.344161 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.344146 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-data" (OuterVolumeSpecName: "istio-data") pod "c667ec74-461c-4ebd-b04c-2f79bf7cbee2" (UID: "c667ec74-461c-4ebd-b04c-2f79bf7cbee2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:40:58.345415 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.345392 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c667ec74-461c-4ebd-b04c-2f79bf7cbee2" (UID: "c667ec74-461c-4ebd-b04c-2f79bf7cbee2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:40:58.345505 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.345416 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-kube-api-access-7zssq" (OuterVolumeSpecName: "kube-api-access-7zssq") pod "c667ec74-461c-4ebd-b04c-2f79bf7cbee2" (UID: "c667ec74-461c-4ebd-b04c-2f79bf7cbee2"). InnerVolumeSpecName "kube-api-access-7zssq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:40:58.345505 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.345468 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-token" (OuterVolumeSpecName: "istio-token") pod "c667ec74-461c-4ebd-b04c-2f79bf7cbee2" (UID: "c667ec74-461c-4ebd-b04c-2f79bf7cbee2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:40:58.345505 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.345498 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c667ec74-461c-4ebd-b04c-2f79bf7cbee2" (UID: "c667ec74-461c-4ebd-b04c-2f79bf7cbee2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 01:40:58.443981 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.443958 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zssq\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-kube-api-access-7zssq\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:40:58.443981 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.443979 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:40:58.444147 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.443989 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:40:58.444147 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.443997 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:40:58.444147 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.444022 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:40:58.444147 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.444030 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:40:58.444147 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.444039 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:40:58.444147 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.444047 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:40:58.444147 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.444054 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c667ec74-461c-4ebd-b04c-2f79bf7cbee2-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:40:58.486610 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.486577 2550 generic.go:356] "Generic (PLEG): container finished" podID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerID="b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1" exitCode=0 Jul 28 01:40:58.486727 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.486642 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" Jul 28 01:40:58.486727 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.486672 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" event={"ID":"c667ec74-461c-4ebd-b04c-2f79bf7cbee2","Type":"ContainerDied","Data":"b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1"} Jul 28 01:40:58.486817 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.486734 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w" event={"ID":"c667ec74-461c-4ebd-b04c-2f79bf7cbee2","Type":"ContainerDied","Data":"0332f1e22a3a306ed59757c8a2ba381635772ab1c8bdf4b93b23b5b63f38ac3f"} Jul 28 01:40:58.487643 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.486757 2550 scope.go:117] "RemoveContainer" containerID="b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1" Jul 28 01:40:58.497608 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.497593 2550 scope.go:117] "RemoveContainer" containerID="67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1" Jul 28 01:40:58.505772 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.505754 2550 scope.go:117] "RemoveContainer" containerID="b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1" Jul 28 01:40:58.506026 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:40:58.505990 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1\": container with ID starting with b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1 not found: ID does not exist" containerID="b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1" Jul 28 01:40:58.506085 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.506057 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1"} err="failed to get container status \"b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1\": rpc error: code = NotFound desc = could not find container \"b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1\": container with ID starting with b2e685a48b235c5652d273c5c1cfaccb04bf07070dd18bdd4ed42f9e8b0cfce1 not found: ID does not exist" Jul 28 01:40:58.506137 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.506088 2550 scope.go:117] "RemoveContainer" containerID="67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1" Jul 28 01:40:58.506328 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:40:58.506308 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1\": container with ID starting with 67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1 not found: ID does not exist" containerID="67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1" Jul 28 01:40:58.506386 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.506337 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1"} err="failed to get container status \"67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1\": rpc error: code = NotFound desc = could not find container \"67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1\": container with ID starting with 67cfe4ee8a1c88df576a1d501377e720b7a2e13be2cd12f5084c71d3d5d0a6d1 not found: ID does not exist" Jul 28 01:40:58.512117 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.512095 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w"] Jul 28 01:40:58.515700 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.515678 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1ce59a-openshift-default-6f9759c656-t9l7w"] Jul 28 01:40:58.784282 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:40:58.784200 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" path="/var/lib/kubelet/pods/c667ec74-461c-4ebd-b04c-2f79bf7cbee2/volumes" Jul 28 01:41:36.825336 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:36.825301 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl"] Jul 28 01:41:36.825863 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:36.825485 2550 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" containerID="cri-o://220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900" gracePeriod=30 Jul 28 01:41:37.450489 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:37.450453 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5775755d57-wbxst_14c48c4e-b078-4cf4-a705-ff81c9db1167/authorino/0.log" Jul 28 01:41:42.086881 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.086829 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:41:42.208237 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208208 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-data\") pod \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " Jul 28 01:41:42.208362 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208241 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istiod-ca-cert\") pod \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " Jul 28 01:41:42.208362 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208263 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-podinfo\") pod \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " Jul 28 01:41:42.208362 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208286 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-envoy\") pod \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " Jul 28 01:41:42.208362 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208347 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-credential-socket\") pod \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " Jul 28 01:41:42.208567 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208378 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-certs\") pod \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " Jul 28 01:41:42.208567 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208408 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8llc\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-kube-api-access-v8llc\") pod \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " Jul 28 01:41:42.208567 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208450 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-token\") pod \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " Jul 28 01:41:42.208567 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208511 2550 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-socket\") pod \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\" (UID: \"cbceaed1-fe79-4ccb-b86b-4e22f62db39f\") " Jul 28 01:41:42.208771 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208692 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbceaed1-fe79-4ccb-b86b-4e22f62db39f" (UID: "cbceaed1-fe79-4ccb-b86b-4e22f62db39f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:41:42.208771 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208641 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbceaed1-fe79-4ccb-b86b-4e22f62db39f" (UID: "cbceaed1-fe79-4ccb-b86b-4e22f62db39f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 01:41:42.208771 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208724 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbceaed1-fe79-4ccb-b86b-4e22f62db39f" (UID: "cbceaed1-fe79-4ccb-b86b-4e22f62db39f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:41:42.208914 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208901 2550 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istiod-ca-cert\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:41:42.208951 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208914 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbceaed1-fe79-4ccb-b86b-4e22f62db39f" (UID: "cbceaed1-fe79-4ccb-b86b-4e22f62db39f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:41:42.208951 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208924 2550 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-credential-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:41:42.208951 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.208940 2550 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-certs\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:41:42.209130 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.209050 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbceaed1-fe79-4ccb-b86b-4e22f62db39f" (UID: "cbceaed1-fe79-4ccb-b86b-4e22f62db39f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:41:42.210477 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.210449 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbceaed1-fe79-4ccb-b86b-4e22f62db39f" (UID: "cbceaed1-fe79-4ccb-b86b-4e22f62db39f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 01:41:42.210714 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.210692 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbceaed1-fe79-4ccb-b86b-4e22f62db39f" (UID: "cbceaed1-fe79-4ccb-b86b-4e22f62db39f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 01:41:42.211104 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.211081 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbceaed1-fe79-4ccb-b86b-4e22f62db39f" (UID: "cbceaed1-fe79-4ccb-b86b-4e22f62db39f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:41:42.211104 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.211090 2550 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-kube-api-access-v8llc" (OuterVolumeSpecName: "kube-api-access-v8llc") pod "cbceaed1-fe79-4ccb-b86b-4e22f62db39f" (UID: "cbceaed1-fe79-4ccb-b86b-4e22f62db39f"). InnerVolumeSpecName "kube-api-access-v8llc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 01:41:42.309679 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.309651 2550 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-data\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:41:42.309679 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.309674 2550 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-podinfo\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:41:42.309820 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.309687 2550 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-envoy\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:41:42.309820 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.309700 2550 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8llc\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-kube-api-access-v8llc\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:41:42.309820 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.309712 2550 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-istio-token\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:41:42.309820 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.309723 2550 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbceaed1-fe79-4ccb-b86b-4e22f62db39f-workload-socket\") on node \"ip-10-0-137-173.ec2.internal\" DevicePath \"\"" Jul 28 01:41:42.753875 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.753843 2550 generic.go:356] "Generic (PLEG): container finished" podID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerID="220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900" exitCode=0 Jul 28 01:41:42.754099 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.753894 2550 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" Jul 28 01:41:42.754099 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.753933 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" event={"ID":"cbceaed1-fe79-4ccb-b86b-4e22f62db39f","Type":"ContainerDied","Data":"220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900"} Jul 28 01:41:42.754099 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.754064 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl" event={"ID":"cbceaed1-fe79-4ccb-b86b-4e22f62db39f","Type":"ContainerDied","Data":"be5c9b1ec55972fcb46798a904b81eb0715375c9a180c1250f40127b71b9fc52"} Jul 28 01:41:42.754307 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.754101 2550 scope.go:117] "RemoveContainer" containerID="220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900" Jul 28 01:41:42.764802 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.764783 2550 scope.go:117] "RemoveContainer" containerID="a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177" Jul 28 01:41:42.773563 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.773542 2550 scope.go:117] "RemoveContainer" containerID="220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900" Jul 28 01:41:42.773824 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:41:42.773805 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900\": container with ID starting with 220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900 not found: ID does not exist" containerID="220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900" Jul 28 01:41:42.773872 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.773832 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900"} err="failed to get container status \"220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900\": rpc error: code = NotFound desc = could not find container \"220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900\": container with ID starting with 220e57f3da7f847d593d47b724b92c1b5c0de5464df0a1267ebfb1ab36f6f900 not found: ID does not exist" Jul 28 01:41:42.773872 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.773847 2550 scope.go:117] "RemoveContainer" containerID="a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177" Jul 28 01:41:42.774132 ip-10-0-137-173 kubenswrapper[2550]: E0728 01:41:42.774114 2550 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177\": container with ID starting with a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177 not found: ID does not exist" containerID="a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177" Jul 28 01:41:42.774185 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.774145 2550 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177"} err="failed to get container status \"a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177\": rpc error: code = NotFound desc = could not find container \"a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177\": container with ID starting with a544853c991bd9a7627579b5274b3b18dd5ca15ce633cfc358b6dc363278a177 not found: ID does not exist" Jul 28 01:41:42.785110 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.785089 2550 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl"] Jul 28 01:41:42.785214 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:42.785119 2550 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-3131e3-openshift-default-5bf8fbd94d-ndlpl"] Jul 28 01:41:43.134612 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:43.134589 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f6f5fc955-dvvj7_d48ab8c7-f5de-4a90-b371-ca85c21aa1a8/manager/0.log" Jul 28 01:41:43.252539 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:43.252518 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sq4gg_788e61d4-30c3-47f3-918f-847e8b4eda03/server/0.log" Jul 28 01:41:43.368982 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:43.368959 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lfxrm_8f186614-7ab6-493e-ae34-a00688b60446/manager/0.log" Jul 28 01:41:43.613289 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:43.613265 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-dmn5h_3e35e475-16c0-49a6-9825-4422336ea1ac/copy-manifests/0.log" Jul 28 01:41:43.620145 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:43.620120 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-dmn5h_3e35e475-16c0-49a6-9825-4422336ea1ac/manager/0.log" Jul 28 01:41:44.113337 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:44.113310 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b489644c9-5ckzx_84308a96-5e6b-40d0-a9f2-e8d7141a1a60/maas-api/0.log" Jul 28 01:41:44.560147 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:44.560117 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-zl8xh_203ed36c-b61e-4775-93c0-1d0c3b95b5c2/postgres/0.log" Jul 28 01:41:44.784081 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:44.784053 2550 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" path="/var/lib/kubelet/pods/cbceaed1-fe79-4ccb-b86b-4e22f62db39f/volumes" Jul 28 01:41:45.356842 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.356815 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2_fc00c633-50f4-4896-879b-e79c77e8ed62/pull/0.log" Jul 28 01:41:45.363631 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.363610 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2_fc00c633-50f4-4896-879b-e79c77e8ed62/extract/0.log" Jul 28 01:41:45.370070 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.370055 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2_fc00c633-50f4-4896-879b-e79c77e8ed62/util/0.log" Jul 28 01:41:45.481864 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.481836 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk_05dd2a2e-4988-4843-a1d9-d9213c6b1a75/util/0.log" Jul 28 01:41:45.489501 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.489478 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk_05dd2a2e-4988-4843-a1d9-d9213c6b1a75/pull/0.log" Jul 28 01:41:45.495189 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.495170 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk_05dd2a2e-4988-4843-a1d9-d9213c6b1a75/extract/0.log" Jul 28 01:41:45.605578 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.605559 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg_081731a9-98e2-4a0f-848e-d728b6424925/util/0.log" Jul 28 01:41:45.611704 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.611654 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg_081731a9-98e2-4a0f-848e-d728b6424925/pull/0.log" Jul 28 01:41:45.618569 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.618541 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg_081731a9-98e2-4a0f-848e-d728b6424925/extract/0.log" Jul 28 01:41:45.734598 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.734575 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr_08c23cc5-be9b-436a-9a9a-04352f7b4714/util/0.log" Jul 28 01:41:45.740731 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.740709 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr_08c23cc5-be9b-436a-9a9a-04352f7b4714/pull/0.log" Jul 28 01:41:45.746584 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.746562 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr_08c23cc5-be9b-436a-9a9a-04352f7b4714/extract/0.log" Jul 28 01:41:45.863390 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.863328 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5775755d57-wbxst_14c48c4e-b078-4cf4-a705-ff81c9db1167/authorino/0.log" Jul 28 01:41:45.982353 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:45.982330 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qhvjx_70f6a41b-664a-489e-9a9e-4236e881af27/manager/0.log" Jul 28 01:41:46.095173 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:46.095148 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zdh6v_c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85/manager/0.log" Jul 28 01:41:46.207474 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:46.207402 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bb2jn_c0787303-0655-4bf8-87a9-92c6c7ad398c/kuadrant-console-plugin/0.log" Jul 28 01:41:46.329065 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:46.329042 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ph7q6_fb01dab4-37cc-4f5b-8c7c-e873bb9c84af/registry-server/0.log" Jul 28 01:41:47.044593 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:47.044560 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n_2034427d-6a25-41c3-9aeb-5e1d4c50f5ec/istio-proxy/0.log" Jul 28 01:41:47.516459 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:47.516395 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9qvnp_d26077a5-940c-4f71-9834-a877a480ba27/istio-proxy/1.log" Jul 28 01:41:47.636542 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:47.636517 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-twl5k_c644166a-c927-4dc1-b046-30b8097b10af/payload-pre-processing/0.log" Jul 28 01:41:48.229055 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:48.229027 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-whk8b_11199715-b494-44fe-bb7b-0cb83e69578e/main/0.log" Jul 28 01:41:48.459916 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:48.459890 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zp9vw_b3616050-ec5a-46d5-9c16-731518c7047d/main/0.log" Jul 28 01:41:48.571308 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:48.571287 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fwd7sk_d96fa59c-be24-476e-b476-fb81598366d0/main/0.log" Jul 28 01:41:57.536869 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:57.536844 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-79x2c_a8abbb14-25bb-42e2-9fb5-0ad1c4f6ecd7/global-pull-secret-syncer/0.log" Jul 28 01:41:57.693905 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:57.693880 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-svcsw_4dfca9b2-d814-4d0f-8754-eec2a38a19fe/konnectivity-agent/0.log" Jul 28 01:41:57.738471 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:41:57.738446 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-173.ec2.internal_4f1301d4be0daa7e23534036d6f77d33/haproxy/0.log" Jul 28 01:42:02.437387 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.437352 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2_fc00c633-50f4-4896-879b-e79c77e8ed62/extract/0.log" Jul 28 01:42:02.463778 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.463752 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2_fc00c633-50f4-4896-879b-e79c77e8ed62/util/0.log" Jul 28 01:42:02.484432 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.484399 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2r2_fc00c633-50f4-4896-879b-e79c77e8ed62/pull/0.log" Jul 28 01:42:02.510963 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.510938 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk_05dd2a2e-4988-4843-a1d9-d9213c6b1a75/extract/0.log" Jul 28 01:42:02.535774 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.535753 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk_05dd2a2e-4988-4843-a1d9-d9213c6b1a75/util/0.log" Jul 28 01:42:02.559852 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.559836 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0wfzvk_05dd2a2e-4988-4843-a1d9-d9213c6b1a75/pull/0.log" Jul 28 01:42:02.588532 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.588514 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg_081731a9-98e2-4a0f-848e-d728b6424925/extract/0.log" Jul 28 01:42:02.611054 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.611033 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg_081731a9-98e2-4a0f-848e-d728b6424925/util/0.log" Jul 28 01:42:02.633334 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.633311 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73hq6gg_081731a9-98e2-4a0f-848e-d728b6424925/pull/0.log" Jul 28 01:42:02.659522 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.659503 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr_08c23cc5-be9b-436a-9a9a-04352f7b4714/extract/0.log" Jul 28 01:42:02.681187 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.681157 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr_08c23cc5-be9b-436a-9a9a-04352f7b4714/util/0.log" Jul 28 01:42:02.704411 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:02.704361 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef15rgxr_08c23cc5-be9b-436a-9a9a-04352f7b4714/pull/0.log" Jul 28 01:42:03.057136 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:03.057095 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5775755d57-wbxst_14c48c4e-b078-4cf4-a705-ff81c9db1167/authorino/0.log" Jul 28 01:42:03.113755 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:03.113728 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qhvjx_70f6a41b-664a-489e-9a9e-4236e881af27/manager/0.log" Jul 28 01:42:03.140790 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:03.140770 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-zdh6v_c5cc8e8d-b557-4b87-a9df-d6c9b8cfaa85/manager/0.log" Jul 28 01:42:03.169950 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:03.169933 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bb2jn_c0787303-0655-4bf8-87a9-92c6c7ad398c/kuadrant-console-plugin/0.log" Jul 28 01:42:03.215340 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:03.215317 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-ph7q6_fb01dab4-37cc-4f5b-8c7c-e873bb9c84af/registry-server/0.log" Jul 28 01:42:05.132788 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.132760 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f81656e6-540d-4544-8dbf-ce0693bc08e1/alertmanager/0.log" Jul 28 01:42:05.162086 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.162060 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f81656e6-540d-4544-8dbf-ce0693bc08e1/config-reloader/0.log" Jul 28 01:42:05.189177 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.189128 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f81656e6-540d-4544-8dbf-ce0693bc08e1/kube-rbac-proxy-web/0.log" Jul 28 01:42:05.210349 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.210320 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f81656e6-540d-4544-8dbf-ce0693bc08e1/kube-rbac-proxy/0.log" Jul 28 01:42:05.233822 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.233802 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f81656e6-540d-4544-8dbf-ce0693bc08e1/kube-rbac-proxy-metric/0.log" Jul 28 01:42:05.257602 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.257581 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f81656e6-540d-4544-8dbf-ce0693bc08e1/prom-label-proxy/0.log" Jul 28 01:42:05.279002 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.278986 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f81656e6-540d-4544-8dbf-ce0693bc08e1/init-config-reloader/0.log" Jul 28 01:42:05.450357 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.450299 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-x2d5r_0f8733cf-36a4-49d4-b534-b13e4e6b0d4c/monitoring-plugin/0.log" Jul 28 01:42:05.626688 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.626667 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wn7mt_a106d45c-6d50-4df5-ae35-fce07bd9bf73/node-exporter/0.log" Jul 28 01:42:05.649182 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.649156 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wn7mt_a106d45c-6d50-4df5-ae35-fce07bd9bf73/kube-rbac-proxy/0.log" Jul 28 01:42:05.673535 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.673516 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wn7mt_a106d45c-6d50-4df5-ae35-fce07bd9bf73/init-textfile/0.log" Jul 28 01:42:05.978524 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.978490 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-478rq_ad4642d9-bfe2-443e-b139-444ae3f216ce/prometheus-operator/0.log" Jul 28 01:42:05.995564 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:05.995522 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-478rq_ad4642d9-bfe2-443e-b139-444ae3f216ce/kube-rbac-proxy/0.log" Jul 28 01:42:06.052386 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:06.052357 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-684d6665f8-7qh58_bb08d8bb-6c62-47e4-9c7c-826f5bc65920/telemeter-client/0.log" Jul 28 01:42:06.073135 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:06.073096 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-684d6665f8-7qh58_bb08d8bb-6c62-47e4-9c7c-826f5bc65920/reload/0.log" Jul 28 01:42:06.093629 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:06.093609 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-684d6665f8-7qh58_bb08d8bb-6c62-47e4-9c7c-826f5bc65920/kube-rbac-proxy/0.log" Jul 28 01:42:06.125748 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:06.125710 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777cb9787d-b2w6v_25703e22-d2c7-40d3-8dcc-2478f30460ec/thanos-query/0.log" Jul 28 01:42:06.146587 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:06.146567 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777cb9787d-b2w6v_25703e22-d2c7-40d3-8dcc-2478f30460ec/kube-rbac-proxy-web/0.log" Jul 28 01:42:06.172023 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:06.171987 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777cb9787d-b2w6v_25703e22-d2c7-40d3-8dcc-2478f30460ec/kube-rbac-proxy/0.log" Jul 28 01:42:06.194442 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:06.194424 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777cb9787d-b2w6v_25703e22-d2c7-40d3-8dcc-2478f30460ec/prom-label-proxy/0.log" Jul 28 01:42:06.216378 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:06.216355 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777cb9787d-b2w6v_25703e22-d2c7-40d3-8dcc-2478f30460ec/kube-rbac-proxy-rules/0.log" Jul 28 01:42:06.237531 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:06.237486 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-777cb9787d-b2w6v_25703e22-d2c7-40d3-8dcc-2478f30460ec/kube-rbac-proxy-metrics/0.log" Jul 28 01:42:07.607433 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607399 2550 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq"] Jul 28 01:42:07.607834 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607782 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerName="istio-proxy" Jul 28 01:42:07.607834 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607793 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerName="istio-proxy" Jul 28 01:42:07.607834 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607799 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerName="istio-proxy" Jul 28 01:42:07.607834 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607805 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerName="istio-proxy" Jul 28 01:42:07.607834 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607824 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" Jul 28 01:42:07.607834 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607830 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" Jul 28 01:42:07.607834 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607836 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" Jul 28 01:42:07.608112 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607842 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" Jul 28 01:42:07.608112 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607850 2550 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c" containerName="registry-server" Jul 28 01:42:07.608112 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607855 2550 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c" containerName="registry-server" Jul 28 01:42:07.608112 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607927 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" Jul 28 01:42:07.608112 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607934 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="c667ec74-461c-4ebd-b04c-2f79bf7cbee2" containerName="istio-proxy" Jul 28 01:42:07.608112 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607940 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="66382a3e-6018-4bb0-bcd4-233f7986b493" containerName="istio-proxy" Jul 28 01:42:07.608112 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607947 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbceaed1-fe79-4ccb-b86b-4e22f62db39f" containerName="istio-proxy" Jul 28 01:42:07.608112 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.607955 2550 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9af9b7d-f18d-4d1b-a5e2-04cdf01b606c" containerName="registry-server" Jul 28 01:42:07.611202 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.611186 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.614262 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.614238 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2v9r2\"/\"kube-root-ca.crt\"" Jul 28 01:42:07.614856 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.614833 2550 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2v9r2\"/\"default-dockercfg-d6gzj\"" Jul 28 01:42:07.614856 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.614850 2550 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2v9r2\"/\"openshift-service-ca.crt\"" Jul 28 01:42:07.629640 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.629619 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq"] Jul 28 01:42:07.701801 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.701781 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-sys\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.701908 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.701819 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sspm6\" (UniqueName: \"kubernetes.io/projected/90b99ce6-726f-499a-aca4-95648b8a8346-kube-api-access-sspm6\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.701908 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.701887 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-podres\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.701982 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.701952 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-lib-modules\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.702058 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.702036 2550 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-proc\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.802960 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.802935 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sspm6\" (UniqueName: \"kubernetes.io/projected/90b99ce6-726f-499a-aca4-95648b8a8346-kube-api-access-sspm6\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.803087 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.802975 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-podres\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.803140 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.803108 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-lib-modules\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.803198 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.803153 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-podres\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.803198 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.803172 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-proc\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.803307 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.803216 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-proc\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.803307 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.803221 2550 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-sys\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.803307 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.803289 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-lib-modules\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.803437 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.803305 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90b99ce6-726f-499a-aca4-95648b8a8346-sys\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.813813 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.813795 2550 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sspm6\" (UniqueName: \"kubernetes.io/projected/90b99ce6-726f-499a-aca4-95648b8a8346-kube-api-access-sspm6\") pod \"perf-node-gather-daemonset-pswrq\" (UID: \"90b99ce6-726f-499a-aca4-95648b8a8346\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:07.920688 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:07.920636 2550 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:08.049660 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:08.049632 2550 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq"] Jul 28 01:42:08.051538 ip-10-0-137-173 kubenswrapper[2550]: W0728 01:42:08.051502 2550 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod90b99ce6_726f_499a_aca4_95648b8a8346.slice/crio-2f49d03986b9d81fa0a6ddeca31c4d075645ee2471ce924b8898f25c77279af3 WatchSource:0}: Error finding container 2f49d03986b9d81fa0a6ddeca31c4d075645ee2471ce924b8898f25c77279af3: Status 404 returned error can't find the container with id 2f49d03986b9d81fa0a6ddeca31c4d075645ee2471ce924b8898f25c77279af3 Jul 28 01:42:08.913109 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:08.913073 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" event={"ID":"90b99ce6-726f-499a-aca4-95648b8a8346","Type":"ContainerStarted","Data":"898c09e9199b3b211d7f8ea46980f75a804437ff102691996f744b2fbb24fbce"} Jul 28 01:42:08.913531 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:08.913124 2550 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" event={"ID":"90b99ce6-726f-499a-aca4-95648b8a8346","Type":"ContainerStarted","Data":"2f49d03986b9d81fa0a6ddeca31c4d075645ee2471ce924b8898f25c77279af3"} Jul 28 01:42:08.913531 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:08.913283 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:08.930171 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:08.930121 2550 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" podStartSLOduration=1.930107754 podStartE2EDuration="1.930107754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:42:08.927770499 +0000 UTC m=+3324.688618519" watchObservedRunningTime="2026-07-28 01:42:08.930107754 +0000 UTC m=+3324.690955837" Jul 28 01:42:10.411973 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:10.411945 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qsxxt_a98530fe-6e13-4892-af34-8d96fb00e3f7/dns/0.log" Jul 28 01:42:10.431720 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:10.431698 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qsxxt_a98530fe-6e13-4892-af34-8d96fb00e3f7/kube-rbac-proxy/0.log" Jul 28 01:42:10.477991 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:10.477968 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wwgrl_a38938d2-ef9d-43ec-91d9-575165164123/dns-node-resolver/0.log" Jul 28 01:42:11.072646 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:11.072614 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4dqfr_7c8da112-99aa-4570-9c43-197db572fa95/node-ca/0.log" Jul 28 01:42:11.975338 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:11.975305 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n_2034427d-6a25-41c3-9aeb-5e1d4c50f5ec/istio-proxy/0.log" Jul 28 01:42:12.214627 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:12.214595 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9qvnp_d26077a5-940c-4f71-9834-a877a480ba27/istio-proxy/0.log" Jul 28 01:42:12.243840 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:12.243760 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9qvnp_d26077a5-940c-4f71-9834-a877a480ba27/istio-proxy/1.log" Jul 28 01:42:12.273977 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:12.273954 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-twl5k_c644166a-c927-4dc1-b046-30b8097b10af/payload-pre-processing/0.log" Jul 28 01:42:13.084279 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:13.084249 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8hn84_f9c96ebe-d995-41ae-a3cb-2ebd07bcdbf4/serve-healthcheck-canary/0.log" Jul 28 01:42:13.672243 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:13.672214 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jzhk9_a8efbf8b-4dbb-45b5-9443-0553e28ffe10/kube-rbac-proxy/0.log" Jul 28 01:42:13.691595 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:13.691569 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jzhk9_a8efbf8b-4dbb-45b5-9443-0553e28ffe10/exporter/0.log" Jul 28 01:42:13.710577 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:13.710555 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jzhk9_a8efbf8b-4dbb-45b5-9443-0553e28ffe10/extractor/0.log" Jul 28 01:42:14.928545 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:14.928521 2550 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pswrq" Jul 28 01:42:16.274975 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:16.274947 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7f6f5fc955-dvvj7_d48ab8c7-f5de-4a90-b371-ca85c21aa1a8/manager/0.log" Jul 28 01:42:16.294870 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:16.294845 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sq4gg_788e61d4-30c3-47f3-918f-847e8b4eda03/server/0.log" Jul 28 01:42:16.365078 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:16.365055 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-lfxrm_8f186614-7ab6-493e-ae34-a00688b60446/manager/0.log" Jul 28 01:42:16.501478 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:16.501449 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-dmn5h_3e35e475-16c0-49a6-9825-4422336ea1ac/manager/0.log" Jul 28 01:42:16.519941 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:16.519922 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-dmn5h_3e35e475-16c0-49a6-9825-4422336ea1ac/copy-manifests/0.log" Jul 28 01:42:17.767287 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:17.767211 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n_2034427d-6a25-41c3-9aeb-5e1d4c50f5ec/istio-proxy/0.log" Jul 28 01:42:17.951214 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:17.951189 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9qvnp_d26077a5-940c-4f71-9834-a877a480ba27/istio-proxy/0.log" Jul 28 01:42:17.979959 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:17.979940 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9qvnp_d26077a5-940c-4f71-9834-a877a480ba27/istio-proxy/1.log" Jul 28 01:42:18.006179 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:18.006157 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-twl5k_c644166a-c927-4dc1-b046-30b8097b10af/payload-pre-processing/0.log" Jul 28 01:42:18.007830 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:18.007806 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f79f7c78b-q2wxd_469d5192-e317-4924-805a-3154a8240376/manager/0.log" Jul 28 01:42:18.039914 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:18.039891 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4st2b_945b8226-b92e-4d96-bcaa-806cd25ca8b1/openshift-lws-operator/0.log" Jul 28 01:42:19.489352 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:19.489274 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-58t8n_2034427d-6a25-41c3-9aeb-5e1d4c50f5ec/istio-proxy/0.log" Jul 28 01:42:19.683419 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:19.683389 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9qvnp_d26077a5-940c-4f71-9834-a877a480ba27/istio-proxy/0.log" Jul 28 01:42:19.710626 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:19.710602 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9qvnp_d26077a5-940c-4f71-9834-a877a480ba27/istio-proxy/1.log" Jul 28 01:42:19.732260 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:19.732238 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-twl5k_c644166a-c927-4dc1-b046-30b8097b10af/payload-pre-processing/0.log" Jul 28 01:42:24.142724 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:24.142697 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6hd56_7858a6fd-db8f-4492-b847-d19cec86528d/kube-multus/0.log" Jul 28 01:42:24.174824 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:24.174801 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6sh29_a14ecd2d-fc55-40dd-85fa-6152eb2ce28c/kube-multus-additional-cni-plugins/0.log" Jul 28 01:42:24.198840 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:24.198818 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6sh29_a14ecd2d-fc55-40dd-85fa-6152eb2ce28c/egress-router-binary-copy/0.log" Jul 28 01:42:24.223433 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:24.223414 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6sh29_a14ecd2d-fc55-40dd-85fa-6152eb2ce28c/cni-plugins/0.log" Jul 28 01:42:24.243649 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:24.243630 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6sh29_a14ecd2d-fc55-40dd-85fa-6152eb2ce28c/bond-cni-plugin/0.log" Jul 28 01:42:24.264706 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:24.264685 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6sh29_a14ecd2d-fc55-40dd-85fa-6152eb2ce28c/routeoverride-cni/0.log" Jul 28 01:42:24.287063 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:24.287024 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6sh29_a14ecd2d-fc55-40dd-85fa-6152eb2ce28c/whereabouts-cni-bincopy/0.log" Jul 28 01:42:24.307952 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:24.307932 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6sh29_a14ecd2d-fc55-40dd-85fa-6152eb2ce28c/whereabouts-cni/0.log" Jul 28 01:42:24.756124 ip-10-0-137-173 kubenswrapper[2550]: I0728 01:42:24.756101 2550 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-gddjn_07a0f217-710e-482b-a1bb-a358656d435f/network-metrics-daemon/0.log"