Jul 08 19:58:26.964550 ip-10-0-132-18 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 08 19:58:26.964565 ip-10-0-132-18 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 08 19:58:26.964574 ip-10-0-132-18 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 08 19:58:26.964924 ip-10-0-132-18 systemd[1]: Failed to start Kubernetes Kubelet. Jul 08 19:58:37.057347 ip-10-0-132-18 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 08 19:58:37.057362 ip-10-0-132-18 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 63631ca9435b4c57a9227f6f388f1e23 -- Jul 08 20:00:30.744695 ip-10-0-132-18 systemd[1]: Starting Kubernetes Kubelet... Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.290758 2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295743 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295755 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295758 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295762 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295765 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295768 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295770 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 20:00:31.359942 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295773 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295776 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295778 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295781 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295784 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295786 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295796 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295799 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295802 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295804 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295807 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295811 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295816 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295819 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295822 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295824 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295827 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295830 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295833 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295836 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 20:00:31.361738 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295838 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295841 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295844 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295846 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295849 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295851 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295854 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295856 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295859 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295861 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295864 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295868 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295870 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295873 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295876 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295879 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295881 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295885 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295889 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295893 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 20:00:31.362652 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295896 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295899 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295902 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295905 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295908 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295910 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295913 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295916 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295919 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295922 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295924 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295927 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295930 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295933 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295935 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295938 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295940 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295943 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295945 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295948 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 20:00:31.363547 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295951 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295954 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295956 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295959 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295961 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295965 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295967 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295972 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295975 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295978 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295980 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295983 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295986 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295989 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295991 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295994 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.295997 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296000 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296004 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 20:00:31.365640 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296007 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296009 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296012 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296015 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296017 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296042 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296047 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296051 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296054 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296057 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296059 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296062 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296064 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296778 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296784 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296787 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296790 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296793 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296796 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 20:00:31.366486 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296799 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296801 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296804 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296807 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296810 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296813 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296816 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296818 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296821 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296823 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296826 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296828 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296831 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296833 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296836 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296839 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296842 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296845 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296849 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296852 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 20:00:31.367591 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296855 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296857 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296860 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296863 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296865 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296868 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296870 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296873 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296875 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296878 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296880 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296883 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296885 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296888 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296890 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296893 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296898 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296902 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296905 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296907 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 20:00:31.368494 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296910 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 20:00:31.368506 ip-10-0-132-18 systemd[1]: Started Kubernetes Kubelet. Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296914 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296917 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296920 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296923 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296925 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296928 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296931 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296934 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296937 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296939 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296942 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296945 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296948 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296950 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296953 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296955 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296958 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296961 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 20:00:31.370161 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296964 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296966 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296969 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296972 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296986 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296990 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296992 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296995 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.296998 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297001 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297004 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297007 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297009 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297012 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297014 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297017 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297020 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297023 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297026 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 20:00:31.371049 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297028 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297031 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297033 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297036 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297039 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297041 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297044 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297046 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297049 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297052 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297054 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297057 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297061 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297064 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297066 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297457 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297462 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297465 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297469 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 08 20:00:31.372431 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297473 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297477 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297480 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297483 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297486 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297490 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297493 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297496 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297499 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297501 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297504 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297508 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297511 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297513 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297516 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297518 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297521 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297524 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297526 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297529 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 20:00:31.373385 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297531 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297534 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297537 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297540 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297542 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297558 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297562 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297566 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297569 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297572 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297575 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297578 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297581 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297583 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297586 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297589 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297592 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297596 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297599 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297602 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 20:00:31.374995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297604 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297607 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297609 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297612 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297614 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297617 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297619 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297622 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297625 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297627 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297630 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297633 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297635 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297638 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297641 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297644 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297646 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297649 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297652 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297654 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 20:00:31.375837 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297656 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297659 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297662 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297664 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297667 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297669 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297672 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297674 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297678 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297681 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297684 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297689 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297692 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297694 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297697 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297700 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297702 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297705 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297707 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 20:00:31.377188 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297710 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297712 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297720 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297722 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297725 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297728 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297730 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297733 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297736 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297738 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297740 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297743 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297746 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297748 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297751 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.297754 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297818 2573 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297825 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297830 2573 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297834 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297838 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 20:00:31.378071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297841 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297846 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297851 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297854 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297860 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297865 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297869 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297872 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297875 2573 flags.go:64] FLAG: --cgroup-root="" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297878 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297881 2573 flags.go:64] FLAG: --client-ca-file="" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297884 2573 flags.go:64] FLAG: --cloud-provider="external" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297887 2573 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297893 2573 flags.go:64] FLAG: --cluster-domain="" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297896 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297899 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297902 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297906 2573 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297910 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297913 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297916 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297920 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297923 2573 flags.go:64] FLAG: --contention-profiling="false" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297926 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 20:00:31.379612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297928 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297932 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297934 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297938 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297941 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297944 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297947 2573 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297951 2573 flags.go:64] FLAG: --enable-server="true" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297954 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297958 2573 flags.go:64] FLAG: --event-burst="100" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297961 2573 flags.go:64] FLAG: --event-qps="50" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297964 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297967 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297970 2573 flags.go:64] FLAG: --eviction-hard="" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297974 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297977 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297980 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297983 2573 flags.go:64] FLAG: --eviction-soft="" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297985 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297988 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297991 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297995 2573 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.297998 2573 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298001 2573 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298004 2573 flags.go:64] FLAG: --feature-gates="" Jul 08 20:00:31.380734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298007 2573 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298011 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298014 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298017 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298020 2573 flags.go:64] FLAG: --healthz-port="10248" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298023 2573 flags.go:64] FLAG: --help="false" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298026 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-132-18.ec2.internal" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298029 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298031 2573 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298034 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298038 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298041 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298044 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298047 2573 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298050 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298053 2573 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298056 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298059 2573 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298062 2573 flags.go:64] FLAG: --kube-reserved="" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298065 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298068 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298071 2573 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298074 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298077 2573 flags.go:64] FLAG: --lock-file="" Jul 08 20:00:31.382925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298080 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298082 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298085 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298090 2573 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298094 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298097 2573 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298100 2573 flags.go:64] FLAG: --logging-format="text" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298103 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298106 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298108 2573 flags.go:64] FLAG: --manifest-url="" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298111 2573 flags.go:64] FLAG: --manifest-url-header="" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298116 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298118 2573 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298122 2573 flags.go:64] FLAG: --max-pods="110" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298130 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298133 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298136 2573 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298138 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298142 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298144 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298147 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298154 2573 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298157 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298160 2573 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 20:00:31.384032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298163 2573 flags.go:64] FLAG: --pod-cidr="" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298166 2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298171 2573 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298174 2573 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298178 2573 flags.go:64] FLAG: --pods-per-core="0" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298181 2573 flags.go:64] FLAG: --port="10250" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298184 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298187 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01d4a1a4aebe5d8d4" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298190 2573 flags.go:64] FLAG: --qos-reserved="" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298193 2573 flags.go:64] FLAG: --read-only-port="10255" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298196 2573 flags.go:64] FLAG: --register-node="true" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298199 2573 flags.go:64] FLAG: --register-with-taints="" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298202 2573 flags.go:64] FLAG: --registry-burst="10" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298206 2573 flags.go:64] FLAG: --registry-qps="5" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298209 2573 flags.go:64] FLAG: --reserved-cpus="" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298212 2573 flags.go:64] FLAG: --reserved-memory="" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298216 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298219 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298222 2573 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298225 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298228 2573 flags.go:64] FLAG: --runonce="false" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298231 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298235 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298239 2573 flags.go:64] FLAG: --seccomp-default="false" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298242 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 20:00:31.385133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298244 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298247 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298250 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298253 2573 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298256 2573 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298259 2573 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298262 2573 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298264 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298267 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298270 2573 flags.go:64] FLAG: --system-cgroups="" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298273 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298278 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298281 2573 flags.go:64] FLAG: --tls-cert-file="" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298284 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298288 2573 flags.go:64] FLAG: --tls-min-version="" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298291 2573 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298293 2573 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298296 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298299 2573 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298302 2573 flags.go:64] FLAG: --v="2" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298306 2573 flags.go:64] FLAG: --version="false" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298310 2573 flags.go:64] FLAG: --vmodule="" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298314 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.298317 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300445 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 20:00:31.386225 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300458 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300461 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300465 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300469 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300472 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300475 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300478 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300481 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300484 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300487 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300490 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300493 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300495 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300498 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300501 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300504 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300506 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300509 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300512 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300514 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 20:00:31.387761 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300517 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300519 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300522 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300525 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300527 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300530 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300533 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300535 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300539 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300541 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300544 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300552 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300556 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300558 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300561 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300564 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300566 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300569 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300571 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 20:00:31.388641 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300574 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300579 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300583 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300586 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300589 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300592 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300596 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300600 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300603 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300606 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300609 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300611 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300614 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300617 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300619 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300622 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300625 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300627 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 20:00:31.389953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300630 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300633 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300635 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300639 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300642 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300645 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300648 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300651 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300653 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300656 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300659 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300661 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300664 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300666 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300669 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300672 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300675 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300677 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300682 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300685 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 20:00:31.390825 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300688 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300691 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300693 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300696 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300699 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300701 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300704 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300706 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300709 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300711 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300714 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300717 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300720 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300723 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300725 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300731 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300733 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300736 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300738 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300742 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 20:00:31.392128 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.300744 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.300750 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.307561 2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.307574 2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.307595 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.307601 2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307655 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307660 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307663 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307667 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307670 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307673 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307675 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307678 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307681 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 20:00:31.392995 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307685 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307689 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307693 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307696 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307699 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307701 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307704 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307707 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307711 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307715 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307719 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307722 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307725 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307727 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307730 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307733 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307738 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307741 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307744 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 20:00:31.394076 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307746 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307749 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307752 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307754 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307757 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307759 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307762 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307764 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307767 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307770 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307772 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307776 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307779 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307782 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307785 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307787 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307791 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307794 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307796 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307799 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 20:00:31.394959 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307801 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307804 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307806 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307809 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307812 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307814 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307816 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307819 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307822 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307825 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307829 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307831 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307834 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307837 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307839 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307842 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307844 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307847 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307850 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307852 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 20:00:31.396690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307855 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307857 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307860 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307862 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307865 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307868 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307870 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307873 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307875 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307879 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307882 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307884 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307887 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307889 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307892 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307894 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307897 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307900 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307902 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307905 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 20:00:31.397575 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307907 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307912 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307915 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307917 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307920 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307923 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307925 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307928 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307931 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307933 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.307935 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.307940 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308048 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308054 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308057 2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 20:00:31.398886 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308060 2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308063 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308065 2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308068 2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308070 2573 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308073 2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308075 2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308078 2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308081 2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308083 2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308087 2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308091 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308094 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308097 2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308101 2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308104 2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308107 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308110 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308114 2573 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 20:00:31.399743 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308117 2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308121 2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308125 2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308128 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308130 2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308134 2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308137 2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308140 2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308142 2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308145 2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308148 2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308150 2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308153 2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308156 2573 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308158 2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308161 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308163 2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308166 2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308168 2573 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308171 2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 20:00:31.400622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308174 2573 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308176 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308179 2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308182 2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308184 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308187 2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308189 2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308192 2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308194 2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308197 2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308200 2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308205 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308209 2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308212 2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308214 2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308217 2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308219 2573 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308222 2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308225 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308227 2573 feature_gate.go:328] unrecognized feature gate: Example Jul 08 20:00:31.401501 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308230 2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308232 2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308235 2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308237 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308240 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308242 2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308245 2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308247 2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308250 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308252 2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308255 2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308257 2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308260 2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308262 2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308265 2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308268 2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308270 2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308273 2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308275 2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 20:00:31.402624 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308278 2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308280 2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308283 2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308285 2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308292 2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308296 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308300 2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308302 2573 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308305 2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308307 2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308310 2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308313 2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308315 2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308318 2573 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308321 2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308338 2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308343 2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 20:00:31.403499 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:31.308346 2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.308351 2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.308449 2573 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.311342 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.312906 2573 server.go:1019] "Starting client certificate rotation" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.313010 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.313044 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.315995 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.321190 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.334890 2573 log.go:25] "Validated CRI v1 runtime API" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.337125 2573 log.go:25] "Validated CRI v1 image API" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.338380 2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 20:00:31.404163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.341723 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7f1ae362-8084-4a54-a4c8-6dc22ba80554:/dev/nvme0n1p3 bd0b3add-908d-4fa7-9d15-aaeefa818f3b:/dev/nvme0n1p4] Jul 08 20:00:31.404817 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.341739 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 20:00:31.404817 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.346898 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.347080 2573 manager.go:216] Machine: {Timestamp:2026-07-08 20:00:31.346397908 +0000 UTC m=+0.463396923 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100733 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec232dea43c33888d0d0786e713f4939 SystemUUID:ec232dea-43c3-3888-d0d0-786e713f4939 BootID:63631ca9-435b-4c57-a922-7f6f388f1e23 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:59:0a:80:9a:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:59:0a:80:9a:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:be:d0:ae:c6:d7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.347211 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.347304 2573 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.351062 2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.351087 2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-18.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.351764 2573 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.351774 2573 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.351786 2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.351798 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.353171 2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.356474 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.356596 2573 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.356626 2573 kubelet.go:486] "Attempting to sync node with API server" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.356634 2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.356648 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.356657 2573 kubelet.go:398] "Adding apiserver pod source" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.356666 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 20:00:32.595360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.357307 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.357321 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.360209 2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.362110 2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.362140 2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363498 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363512 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363518 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363524 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363532 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363538 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363546 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363552 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363558 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363564 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363576 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.363581 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.364447 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.364457 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.367587 2573 server.go:1268] "Started kubelet" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.368376 2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.369092 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.369200 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.370527 2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.370589 2573 server.go:310] "Adding debug handlers to kubelet server" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.372776 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-18.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:31.372773 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-18.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:31.375627 2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.377354 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.378366 2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:31.378673 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379657 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379671 2573 factory.go:55] Registering systemd factory Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379681 2573 factory.go:223] Registration of the systemd container factory successfully Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379763 2573 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379774 2573 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379802 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379866 2573 factory.go:153] Registering CRI-O factory Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379879 2573 factory.go:223] Registration of the crio container factory successfully Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379880 2573 reconstruct.go:97] "Volume reconstruction finished" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379888 2573 reconciler.go:29] "Reconciler: start to sync state" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379899 2573 factory.go:103] Registering Raw factory Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.379911 2573 manager.go:1196] Started watching for new ooms in manager Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.412581 2573 manager.go:349] Starting recovery of all containers Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.412697 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v5dl7" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.412715 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v5dl7" Jul 08 20:00:32.596498 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:31.414285 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:31.414910 2573 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-18.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:31.415563 2573 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:31.415321 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-18.ec2.internal.18c068efc0118137 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-18.ec2.internal,UID:ip-10-0-132-18.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-18.ec2.internal,},FirstTimestamp:2026-07-08 20:00:31.367561527 +0000 UTC m=+0.484560541,LastTimestamp:2026-07-08 20:00:31.367561527 +0000 UTC m=+0.484560541,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-18.ec2.internal,}" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.437514 2573 manager.go:354] Recovery completed Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.442886 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.444878 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientMemory" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.444898 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.444908 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientPID" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.445544 2573 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.445552 2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.445565 2573 state_mem.go:36] "Initialized new in-memory state store" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.446535 2573 policy_none.go:49] "None policy: Start" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.446546 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.446555 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.447503 2573 policy_none.go:47] "Start" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.448079 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.448854 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientMemory" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.448870 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:31.448879 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientPID" Jul 08 20:00:32.599105 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.599043 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.631293 2573 manager.go:337] "Starting Device Plugin manager" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.631349 2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.631360 2573 server.go:87] "Starting device plugin registration server" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.631601 2573 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.631611 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.631703 2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.631788 2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.631798 2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.632409 2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.632447 2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.732139 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.733183 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientMemory" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.733244 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.733256 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientPID" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.733283 2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-18.ec2.internal" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.740410 2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-18.ec2.internal" Jul 08 20:00:32.751261 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.740427 2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-18.ec2.internal\": node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:32.857232 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.763743 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:32.857232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.778683 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 20:00:32.857232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.779974 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 20:00:32.857232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.780007 2573 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 20:00:32.857232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.780061 2573 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 20:00:32.857232 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.780092 2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 20:00:32.857232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.790635 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.864375 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.880516 2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal"] Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.880609 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.883005 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientMemory" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.883032 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.883046 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientPID" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.884539 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.884706 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.884731 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.886974 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientMemory" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.887204 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientMemory" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.887210 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.887222 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.887225 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientPID" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.887255 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientPID" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.888563 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.888582 2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.889396 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientMemory" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.889419 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.889433 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeHasSufficientPID" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.902732 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e6daa43de5249e7297b90bca40b8798-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal\" (UID: \"9e6daa43de5249e7297b90bca40b8798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.902957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e6daa43de5249e7297b90bca40b8798-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal\" (UID: \"9e6daa43de5249e7297b90bca40b8798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:32.903184 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/811aeb70eccf3bdd8a847e099989ebe9-config\") pod \"kube-apiserver-proxy-ip-10-0-132-18.ec2.internal\" (UID: \"811aeb70eccf3bdd8a847e099989ebe9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal" Jul 08 20:00:32.951709 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.940917 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-18.ec2.internal\" not found" node="ip-10-0-132-18.ec2.internal" Jul 08 20:00:32.954778 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.952898 2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-18.ec2.internal\" not found" node="ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.026076 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:32.964650 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:33.026076 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.004159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e6daa43de5249e7297b90bca40b8798-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal\" (UID: \"9e6daa43de5249e7297b90bca40b8798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.026076 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.004187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e6daa43de5249e7297b90bca40b8798-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal\" (UID: \"9e6daa43de5249e7297b90bca40b8798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.026076 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.004208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/811aeb70eccf3bdd8a847e099989ebe9-config\") pod \"kube-apiserver-proxy-ip-10-0-132-18.ec2.internal\" (UID: \"811aeb70eccf3bdd8a847e099989ebe9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.026076 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.004237 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/811aeb70eccf3bdd8a847e099989ebe9-config\") pod \"kube-apiserver-proxy-ip-10-0-132-18.ec2.internal\" (UID: \"811aeb70eccf3bdd8a847e099989ebe9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.026076 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.004274 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9e6daa43de5249e7297b90bca40b8798-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal\" (UID: \"9e6daa43de5249e7297b90bca40b8798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.026076 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.004306 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9e6daa43de5249e7297b90bca40b8798-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal\" (UID: \"9e6daa43de5249e7297b90bca40b8798\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.105071 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:33.065656 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:33.165930 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:33.165899 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:33.244376 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.244346 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.255897 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.255882 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.266436 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:33.266419 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:33.366984 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:33.366919 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:33.432359 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.432337 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 20:00:33.432486 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.432468 2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 20:00:33.467639 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:33.467607 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:33.535643 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.535616 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 19:55:32 +0000 UTC" deadline="2028-02-05 19:35:17.580562158 +0000 UTC" Jul 08 20:00:33.535643 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.535641 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13847h34m44.044925653s" Jul 08 20:00:33.567731 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:33.567701 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:33.588351 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.588327 2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 20:00:33.668476 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:33.668415 2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-18.ec2.internal\" not found" Jul 08 20:00:33.686095 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.686079 2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 20:00:33.700050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.700034 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.715278 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.715260 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 20:00:33.716345 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.716333 2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal" Jul 08 20:00:33.726289 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.726274 2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 20:00:33.927323 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:33.927247 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6daa43de5249e7297b90bca40b8798.slice/crio-a5680f73e182d61950740757ceaff02b59108372a957db9ae8d5a60624dc9403 WatchSource:0}: Error finding container a5680f73e182d61950740757ceaff02b59108372a957db9ae8d5a60624dc9403: Status 404 returned error can't find the container with id a5680f73e182d61950740757ceaff02b59108372a957db9ae8d5a60624dc9403 Jul 08 20:00:33.932425 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:33.932406 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:00:33.962121 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:33.962098 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod811aeb70eccf3bdd8a847e099989ebe9.slice/crio-c716d4be0714772c08ef0cf403996789e9f51f5fb0b92b8bf6c6a2b5854c60a2 WatchSource:0}: Error finding container c716d4be0714772c08ef0cf403996789e9f51f5fb0b92b8bf6c6a2b5854c60a2: Status 404 returned error can't find the container with id c716d4be0714772c08ef0cf403996789e9f51f5fb0b92b8bf6c6a2b5854c60a2 Jul 08 20:00:34.065568 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.065544 2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 20:00:34.477827 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.477806 2573 apiserver.go:52] "Watching apiserver" Jul 08 20:00:34.486130 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.486113 2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 20:00:34.486515 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.486497 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-54tqn","kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal","openshift-dns/node-resolver-w5dc5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal","openshift-multus/network-metrics-daemon-5glh6","openshift-ovn-kubernetes/ovnkube-node-kdwjg","kube-system/global-pull-secret-syncer-4jt2s","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h","openshift-cluster-node-tuning-operator/tuned-8kdxf","openshift-image-registry/node-ca-5tkjn","openshift-multus/multus-additional-cni-plugins-zjhks","openshift-multus/multus-sz827","openshift-network-diagnostics/network-check-target-8jwsw","openshift-network-operator/iptables-alerter-ghlzv"] Jul 08 20:00:34.488398 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.488378 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.489391 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.489369 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz827" Jul 08 20:00:34.490445 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.490428 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.491016 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.490994 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 20:00:34.491118 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.491100 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 20:00:34.491270 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.491252 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 20:00:34.491331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.491298 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-plmqf\"" Jul 08 20:00:34.491596 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.491580 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:34.491720 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.491706 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 20:00:34.491960 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.491935 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 20:00:34.492516 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.492500 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 20:00:34.492701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.492600 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sf64l\"" Jul 08 20:00:34.492701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.492620 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 20:00:34.492701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.492628 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 20:00:34.492877 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.492818 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.493255 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.493238 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 20:00:34.493339 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.493293 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dgm57\"" Jul 08 20:00:34.494024 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.494006 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.494923 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.494822 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 20:00:34.495583 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.495224 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 20:00:34.495583 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.495325 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f68r2\"" Jul 08 20:00:34.496554 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.496074 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 20:00:34.496554 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.496173 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 20:00:34.496554 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.496332 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n6x45\"" Jul 08 20:00:34.496554 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.496421 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 20:00:34.497114 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.496720 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.497986 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.497849 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 20:00:34.498088 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.498000 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 20:00:34.498236 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.498218 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:34.498580 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.498555 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:34.498926 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.498906 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 20:00:34.499003 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.498911 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 20:00:34.499814 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.499795 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 20:00:34.499913 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.499850 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2hl6h\"" Jul 08 20:00:34.499913 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.499864 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 20:00:34.500018 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.499914 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:34.500070 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.500028 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:34.500070 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.500050 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 20:00:34.500070 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.500061 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ft2ll\"" Jul 08 20:00:34.500445 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.500422 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 20:00:34.501136 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.501121 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.502364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.502349 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:34.502464 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.502442 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:34.503296 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.503281 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 20:00:34.503412 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.503395 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 20:00:34.503670 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.503654 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.503814 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.503795 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 20:00:34.503886 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.503834 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d4rbf\"" Jul 08 20:00:34.507002 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.506979 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 20:00:34.507101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.507042 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 20:00:34.507101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.507088 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 20:00:34.507458 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.507443 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6dwkq\"" Jul 08 20:00:34.515728 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.515709 2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 20:00:34.521202 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-node-log\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.521284 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.521284 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521226 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.521284 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vfr\" (UniqueName: \"kubernetes.io/projected/390ebffc-4753-4b01-9f86-8116898f4c36-kube-api-access-v4vfr\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.521284 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521259 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-kubelet-config\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:34.521438 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-dbus\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:34.521474 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521432 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-multus-cni-dir\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.521474 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-run-netns\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.521547 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521501 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd247e9d-54a0-4db5-a1a7-404353527533-multus-daemon-config\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.521547 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-run-multus-certs\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.521611 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521554 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnhck\" (UniqueName: \"kubernetes.io/projected/33019580-9b46-4bda-a0dd-3687d6e3026b-kube-api-access-cnhck\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.521611 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521578 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f05057eb-dc4d-45bd-adf9-1e20fdb9c027-host\") pod \"node-ca-5tkjn\" (UID: \"f05057eb-dc4d-45bd-adf9-1e20fdb9c027\") " pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.521611 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521601 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd247e9d-54a0-4db5-a1a7-404353527533-cni-binary-copy\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.521745 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-var-lib-cni-multus\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.521745 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521644 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f05057eb-dc4d-45bd-adf9-1e20fdb9c027-serviceca\") pod \"node-ca-5tkjn\" (UID: \"f05057eb-dc4d-45bd-adf9-1e20fdb9c027\") " pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.521745 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-run-systemd\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.521745 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521717 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtdj\" (UniqueName: \"kubernetes.io/projected/2cb1ff8c-6dc6-4981-a487-b73961dbd578-kube-api-access-wvtdj\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:34.521904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521744 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-os-release\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.521904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521766 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33019580-9b46-4bda-a0dd-3687d6e3026b-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.521904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521791 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-host\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.521904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-cnibin\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.521904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-systemd\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.521904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521883 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.522129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521905 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-run-openvswitch\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.522129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-cni-bin\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.522129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:34.522129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.521976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-os-release\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.522129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-multus-socket-dir-parent\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.522129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ffn\" (UniqueName: \"kubernetes.io/projected/bd247e9d-54a0-4db5-a1a7-404353527533-kube-api-access-m7ffn\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.522129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522079 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-cnibin\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.522129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522111 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/33019580-9b46-4bda-a0dd-3687d6e3026b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-sys-fs\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522152 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stx5\" (UniqueName: \"kubernetes.io/projected/a35b692c-4d5f-4e7a-addd-2ed017343ab4-kube-api-access-2stx5\") pod \"node-resolver-w5dc5\" (UID: \"a35b692c-4d5f-4e7a-addd-2ed017343ab4\") " pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522166 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-multus-conf-dir\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522180 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-modprobe-d\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-kubernetes\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7m2\" (UniqueName: \"kubernetes.io/projected/09ed8740-dbe9-47b1-b6b4-d843a84df087-kube-api-access-ct7m2\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522236 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-kubelet\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-etc-openvswitch\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522281 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-hostroot\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522299 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-sysconfig\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522331 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-log-socket\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-etc-kubernetes\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522359 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.522384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-run\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp2n6\" (UniqueName: \"kubernetes.io/projected/527467d3-2dd6-4220-938d-33e2b5609951-kube-api-access-jp2n6\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/390ebffc-4753-4b01-9f86-8116898f4c36-env-overrides\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522433 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-run-k8s-cni-cncf-io\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522459 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-sysctl-conf\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-sys\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522523 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-system-cni-dir\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522543 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba630e65-9372-469b-8b3e-9d88d405720e-agent-certs\") pod \"konnectivity-agent-54tqn\" (UID: \"ba630e65-9372-469b-8b3e-9d88d405720e\") " pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522571 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba630e65-9372-469b-8b3e-9d88d405720e-konnectivity-ca\") pod \"konnectivity-agent-54tqn\" (UID: \"ba630e65-9372-469b-8b3e-9d88d405720e\") " pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522591 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/527467d3-2dd6-4220-938d-33e2b5609951-tmp\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522606 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-registration-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522633 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srk77\" (UniqueName: \"kubernetes.io/projected/f05057eb-dc4d-45bd-adf9-1e20fdb9c027-kube-api-access-srk77\") pod \"node-ca-5tkjn\" (UID: \"f05057eb-dc4d-45bd-adf9-1e20fdb9c027\") " pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522657 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-slash\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522672 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-var-lib-openvswitch\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-var-lib-cni-bin\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522725 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-system-cni-dir\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.522802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-sysctl-d\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522756 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-var-lib-kubelet\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522772 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-run-netns\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522794 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/390ebffc-4753-4b01-9f86-8116898f4c36-ovnkube-script-lib\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a35b692c-4d5f-4e7a-addd-2ed017343ab4-hosts-file\") pod \"node-resolver-w5dc5\" (UID: \"a35b692c-4d5f-4e7a-addd-2ed017343ab4\") " pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522868 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35b692c-4d5f-4e7a-addd-2ed017343ab4-tmp-dir\") pod \"node-resolver-w5dc5\" (UID: \"a35b692c-4d5f-4e7a-addd-2ed017343ab4\") " pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522887 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-lib-modules\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-socket-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522926 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-systemd-units\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522945 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-run-ovn\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522959 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-cni-netd\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/390ebffc-4753-4b01-9f86-8116898f4c36-ovn-node-metrics-cert\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.522993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-var-lib-kubelet\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.523029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/33019580-9b46-4bda-a0dd-3687d6e3026b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.523043 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/527467d3-2dd6-4220-938d-33e2b5609951-etc-tuned\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.523273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.523070 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-device-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.523795 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.523090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-etc-selinux\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.523795 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.523110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/390ebffc-4753-4b01-9f86-8116898f4c36-ovnkube-config\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.556963 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.556940 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t5jwh" Jul 08 20:00:34.566382 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.566355 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t5jwh" Jul 08 20:00:34.601595 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.601573 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 20:00:34.623277 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623251 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:34.623413 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623286 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-os-release\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.623413 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-multus-socket-dir-parent\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.623413 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623340 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ffn\" (UniqueName: \"kubernetes.io/projected/bd247e9d-54a0-4db5-a1a7-404353527533-kube-api-access-m7ffn\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.623413 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-cnibin\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.623413 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623389 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/33019580-9b46-4bda-a0dd-3687d6e3026b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.623661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-sys-fs\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.623661 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.623445 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:34.623661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623462 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stx5\" (UniqueName: \"kubernetes.io/projected/a35b692c-4d5f-4e7a-addd-2ed017343ab4-kube-api-access-2stx5\") pod \"node-resolver-w5dc5\" (UID: \"a35b692c-4d5f-4e7a-addd-2ed017343ab4\") " pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.623661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-multus-conf-dir\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.623661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623513 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-os-release\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.623661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623531 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-multus-conf-dir\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.623661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623574 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-cnibin\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.623661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623583 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-sys-fs\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.624031 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.623671 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs podName:2cb1ff8c-6dc6-4981-a487-b73961dbd578 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:35.123496309 +0000 UTC m=+3.120513238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs") pod "network-metrics-daemon-5glh6" (UID: "2cb1ff8c-6dc6-4981-a487-b73961dbd578") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:34.624031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623727 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-multus-socket-dir-parent\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.624031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623747 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-modprobe-d\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.624031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-kubernetes\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.624031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7m2\" (UniqueName: \"kubernetes.io/projected/09ed8740-dbe9-47b1-b6b4-d843a84df087-kube-api-access-ct7m2\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.624031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623870 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-kubelet\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.624031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623885 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-modprobe-d\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.624031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623895 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-kubernetes\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.624031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.623893 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-etc-openvswitch\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624087 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/33019580-9b46-4bda-a0dd-3687d6e3026b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624155 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-hostroot\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-sysconfig\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624210 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-log-socket\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624229 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-etc-openvswitch\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624238 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-etc-kubernetes\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-run\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624305 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-hostroot\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp2n6\" (UniqueName: \"kubernetes.io/projected/527467d3-2dd6-4220-938d-33e2b5609951-kube-api-access-jp2n6\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624269 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-kubelet\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/390ebffc-4753-4b01-9f86-8116898f4c36-env-overrides\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624373 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-sysconfig\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624394 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-run-k8s-cni-cncf-io\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.624415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624414 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-etc-kubernetes\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-sysctl-conf\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624453 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-sys\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624470 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-run\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624476 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-system-cni-dir\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624400 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-log-socket\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba630e65-9372-469b-8b3e-9d88d405720e-agent-certs\") pod \"konnectivity-agent-54tqn\" (UID: \"ba630e65-9372-469b-8b3e-9d88d405720e\") " pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624500 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624522 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-run-k8s-cni-cncf-io\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624522 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba630e65-9372-469b-8b3e-9d88d405720e-konnectivity-ca\") pod \"konnectivity-agent-54tqn\" (UID: \"ba630e65-9372-469b-8b3e-9d88d405720e\") " pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/527467d3-2dd6-4220-938d-33e2b5609951-tmp\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624563 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-sys\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624602 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-registration-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624629 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srk77\" (UniqueName: \"kubernetes.io/projected/f05057eb-dc4d-45bd-adf9-1e20fdb9c027-kube-api-access-srk77\") pod \"node-ca-5tkjn\" (UID: \"f05057eb-dc4d-45bd-adf9-1e20fdb9c027\") " pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624650 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-slash\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624659 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-sysctl-conf\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624672 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-var-lib-openvswitch\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624695 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-var-lib-cni-bin\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.625104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624716 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-system-cni-dir\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-system-cni-dir\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624743 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-sysctl-d\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624774 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-slash\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624766 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-var-lib-kubelet\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624819 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-var-lib-cni-bin\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624832 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-var-lib-kubelet\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-run-netns\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624886 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-var-lib-openvswitch\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/390ebffc-4753-4b01-9f86-8116898f4c36-ovnkube-script-lib\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624907 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-run-netns\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624953 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a35b692c-4d5f-4e7a-addd-2ed017343ab4-hosts-file\") pod \"node-resolver-w5dc5\" (UID: \"a35b692c-4d5f-4e7a-addd-2ed017343ab4\") " pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624984 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-system-cni-dir\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625006 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/390ebffc-4753-4b01-9f86-8116898f4c36-env-overrides\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624992 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.624995 2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625047 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a35b692c-4d5f-4e7a-addd-2ed017343ab4-hosts-file\") pod \"node-resolver-w5dc5\" (UID: \"a35b692c-4d5f-4e7a-addd-2ed017343ab4\") " pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625054 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35b692c-4d5f-4e7a-addd-2ed017343ab4-tmp-dir\") pod \"node-resolver-w5dc5\" (UID: \"a35b692c-4d5f-4e7a-addd-2ed017343ab4\") " pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.625928 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.625050 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625150 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-sysctl-d\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625067 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ba630e65-9372-469b-8b3e-9d88d405720e-konnectivity-ca\") pod \"konnectivity-agent-54tqn\" (UID: \"ba630e65-9372-469b-8b3e-9d88d405720e\") " pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.625154 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret podName:938c8e6d-1bc6-432e-a7fd-f1c3da1bf189 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:35.125119324 +0000 UTC m=+3.122136450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret") pod "global-pull-secret-syncer-4jt2s" (UID: "938c8e6d-1bc6-432e-a7fd-f1c3da1bf189") : object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-lib-modules\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625237 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-socket-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-systemd-units\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625284 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-run-ovn\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625321 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-cni-netd\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625354 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35b692c-4d5f-4e7a-addd-2ed017343ab4-tmp-dir\") pod \"node-resolver-w5dc5\" (UID: \"a35b692c-4d5f-4e7a-addd-2ed017343ab4\") " pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625370 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-cni-netd\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625371 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-lib-modules\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625390 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-registration-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625428 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-systemd-units\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625433 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-run-ovn\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625445 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/390ebffc-4753-4b01-9f86-8116898f4c36-ovn-node-metrics-cert\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625466 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-socket-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.626774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625478 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/390ebffc-4753-4b01-9f86-8116898f4c36-ovnkube-script-lib\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-var-lib-kubelet\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/33019580-9b46-4bda-a0dd-3687d6e3026b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/527467d3-2dd6-4220-938d-33e2b5609951-etc-tuned\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-device-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-etc-selinux\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/390ebffc-4753-4b01-9f86-8116898f4c36-ovnkube-config\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625677 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4364e2f1-93ad-425d-af18-8d77880860db-iptables-alerter-script\") pod \"iptables-alerter-ghlzv\" (UID: \"4364e2f1-93ad-425d-af18-8d77880860db\") " pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625687 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-device-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-node-log\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625762 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625810 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-node-log\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625884 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-etc-selinux\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625912 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vfr\" (UniqueName: \"kubernetes.io/projected/390ebffc-4753-4b01-9f86-8116898f4c36-kube-api-access-v4vfr\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-kubelet-config\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:34.627602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625980 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/33019580-9b46-4bda-a0dd-3687d6e3026b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.625983 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-dbus\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-multus-cni-dir\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-run-netns\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626118 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd247e9d-54a0-4db5-a1a7-404353527533-multus-daemon-config\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-run-multus-certs\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626161 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-dbus\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnhck\" (UniqueName: \"kubernetes.io/projected/33019580-9b46-4bda-a0dd-3687d6e3026b-kube-api-access-cnhck\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626195 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprgd\" (UniqueName: \"kubernetes.io/projected/4364e2f1-93ad-425d-af18-8d77880860db-kube-api-access-pprgd\") pod \"iptables-alerter-ghlzv\" (UID: \"4364e2f1-93ad-425d-af18-8d77880860db\") " pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f05057eb-dc4d-45bd-adf9-1e20fdb9c027-host\") pod \"node-ca-5tkjn\" (UID: \"f05057eb-dc4d-45bd-adf9-1e20fdb9c027\") " pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd247e9d-54a0-4db5-a1a7-404353527533-cni-binary-copy\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-var-lib-cni-multus\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f05057eb-dc4d-45bd-adf9-1e20fdb9c027-serviceca\") pod \"node-ca-5tkjn\" (UID: \"f05057eb-dc4d-45bd-adf9-1e20fdb9c027\") " pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626336 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-run-systemd\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626358 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-var-lib-kubelet\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626363 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtdj\" (UniqueName: \"kubernetes.io/projected/2cb1ff8c-6dc6-4981-a487-b73961dbd578-kube-api-access-wvtdj\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:34.628396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626387 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-kubelet-config\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626404 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4364e2f1-93ad-425d-af18-8d77880860db-host-slash\") pod \"iptables-alerter-ghlzv\" (UID: \"4364e2f1-93ad-425d-af18-8d77880860db\") " pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626395 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/390ebffc-4753-4b01-9f86-8116898f4c36-ovnkube-config\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626429 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-os-release\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626437 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-var-lib-cni-multus\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626444 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-multus-cni-dir\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33019580-9b46-4bda-a0dd-3687d6e3026b-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626495 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33019580-9b46-4bda-a0dd-3687d6e3026b-os-release\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626510 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-run-systemd\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626483 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-run-netns\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626645 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f05057eb-dc4d-45bd-adf9-1e20fdb9c027-host\") pod \"node-ca-5tkjn\" (UID: \"f05057eb-dc4d-45bd-adf9-1e20fdb9c027\") " pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626688 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626858 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-host-run-multus-certs\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-host\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-cnibin\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626916 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f05057eb-dc4d-45bd-adf9-1e20fdb9c027-serviceca\") pod \"node-ca-5tkjn\" (UID: \"f05057eb-dc4d-45bd-adf9-1e20fdb9c027\") " pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626929 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-systemd\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.629194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.626981 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-run-openvswitch\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-cni-bin\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627067 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd247e9d-54a0-4db5-a1a7-404353527533-multus-daemon-config\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627074 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-host-cni-bin\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627122 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd247e9d-54a0-4db5-a1a7-404353527533-cnibin\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627123 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-host\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627167 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/390ebffc-4753-4b01-9f86-8116898f4c36-run-openvswitch\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627180 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/09ed8740-dbe9-47b1-b6b4-d843a84df087-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627221 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/527467d3-2dd6-4220-938d-33e2b5609951-etc-systemd\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627253 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33019580-9b46-4bda-a0dd-3687d6e3026b-cni-binary-copy\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.627372 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd247e9d-54a0-4db5-a1a7-404353527533-cni-binary-copy\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.628887 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/527467d3-2dd6-4220-938d-33e2b5609951-tmp\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.629162 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/390ebffc-4753-4b01-9f86-8116898f4c36-ovn-node-metrics-cert\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.629894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.629187 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ba630e65-9372-469b-8b3e-9d88d405720e-agent-certs\") pod \"konnectivity-agent-54tqn\" (UID: \"ba630e65-9372-469b-8b3e-9d88d405720e\") " pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:34.630350 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.630206 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/527467d3-2dd6-4220-938d-33e2b5609951-etc-tuned\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.632366 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.632288 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stx5\" (UniqueName: \"kubernetes.io/projected/a35b692c-4d5f-4e7a-addd-2ed017343ab4-kube-api-access-2stx5\") pod \"node-resolver-w5dc5\" (UID: \"a35b692c-4d5f-4e7a-addd-2ed017343ab4\") " pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.632712 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.632694 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ffn\" (UniqueName: \"kubernetes.io/projected/bd247e9d-54a0-4db5-a1a7-404353527533-kube-api-access-m7ffn\") pod \"multus-sz827\" (UID: \"bd247e9d-54a0-4db5-a1a7-404353527533\") " pod="openshift-multus/multus-sz827" Jul 08 20:00:34.633133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.633109 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7m2\" (UniqueName: \"kubernetes.io/projected/09ed8740-dbe9-47b1-b6b4-d843a84df087-kube-api-access-ct7m2\") pod \"aws-ebs-csi-driver-node-9dt8h\" (UID: \"09ed8740-dbe9-47b1-b6b4-d843a84df087\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.633552 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.633529 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srk77\" (UniqueName: \"kubernetes.io/projected/f05057eb-dc4d-45bd-adf9-1e20fdb9c027-kube-api-access-srk77\") pod \"node-ca-5tkjn\" (UID: \"f05057eb-dc4d-45bd-adf9-1e20fdb9c027\") " pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.634486 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.634468 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp2n6\" (UniqueName: \"kubernetes.io/projected/527467d3-2dd6-4220-938d-33e2b5609951-kube-api-access-jp2n6\") pod \"tuned-8kdxf\" (UID: \"527467d3-2dd6-4220-938d-33e2b5609951\") " pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.635071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.635049 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnhck\" (UniqueName: \"kubernetes.io/projected/33019580-9b46-4bda-a0dd-3687d6e3026b-kube-api-access-cnhck\") pod \"multus-additional-cni-plugins-zjhks\" (UID: \"33019580-9b46-4bda-a0dd-3687d6e3026b\") " pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.635617 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.635596 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtdj\" (UniqueName: \"kubernetes.io/projected/2cb1ff8c-6dc6-4981-a487-b73961dbd578-kube-api-access-wvtdj\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:34.635782 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.635766 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vfr\" (UniqueName: \"kubernetes.io/projected/390ebffc-4753-4b01-9f86-8116898f4c36-kube-api-access-v4vfr\") pod \"ovnkube-node-kdwjg\" (UID: \"390ebffc-4753-4b01-9f86-8116898f4c36\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.705991 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.705969 2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 20:00:34.728064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.728009 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:34.728064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.728044 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprgd\" (UniqueName: \"kubernetes.io/projected/4364e2f1-93ad-425d-af18-8d77880860db-kube-api-access-pprgd\") pod \"iptables-alerter-ghlzv\" (UID: \"4364e2f1-93ad-425d-af18-8d77880860db\") " pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.728064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.728061 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4364e2f1-93ad-425d-af18-8d77880860db-host-slash\") pod \"iptables-alerter-ghlzv\" (UID: \"4364e2f1-93ad-425d-af18-8d77880860db\") " pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.728212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.728129 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4364e2f1-93ad-425d-af18-8d77880860db-host-slash\") pod \"iptables-alerter-ghlzv\" (UID: \"4364e2f1-93ad-425d-af18-8d77880860db\") " pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.728243 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.728232 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4364e2f1-93ad-425d-af18-8d77880860db-iptables-alerter-script\") pod \"iptables-alerter-ghlzv\" (UID: \"4364e2f1-93ad-425d-af18-8d77880860db\") " pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.728669 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.728651 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4364e2f1-93ad-425d-af18-8d77880860db-iptables-alerter-script\") pod \"iptables-alerter-ghlzv\" (UID: \"4364e2f1-93ad-425d-af18-8d77880860db\") " pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.734593 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.734579 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 20:00:34.734630 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.734597 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 20:00:34.734630 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.734606 2573 projected.go:196] Error preparing data for projected volume kube-api-access-p4cz2 for pod openshift-network-diagnostics/network-check-target-8jwsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:34.734692 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:34.734664 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2 podName:ac575ae2-11c1-4daa-8d9b-4b675f53f832 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:35.23464995 +0000 UTC m=+3.231667069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-p4cz2" (UniqueName: "kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2") pod "network-check-target-8jwsw" (UID: "ac575ae2-11c1-4daa-8d9b-4b675f53f832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:34.736399 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.736385 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprgd\" (UniqueName: \"kubernetes.io/projected/4364e2f1-93ad-425d-af18-8d77880860db-kube-api-access-pprgd\") pod \"iptables-alerter-ghlzv\" (UID: \"4364e2f1-93ad-425d-af18-8d77880860db\") " pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.784159 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.784105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal" event={"ID":"811aeb70eccf3bdd8a847e099989ebe9","Type":"ContainerStarted","Data":"c716d4be0714772c08ef0cf403996789e9f51f5fb0b92b8bf6c6a2b5854c60a2"} Jul 08 20:00:34.785279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.785257 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" event={"ID":"9e6daa43de5249e7297b90bca40b8798","Type":"ContainerStarted","Data":"a5680f73e182d61950740757ceaff02b59108372a957db9ae8d5a60624dc9403"} Jul 08 20:00:34.799514 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.799496 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" Jul 08 20:00:34.805778 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.805759 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz827" Jul 08 20:00:34.809614 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:34.809593 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ed8740_dbe9_47b1_b6b4_d843a84df087.slice/crio-7e41a7b51ebeb4d05f682d462ea87e6fbc744e3b050fe2d85eb4d0c959c78cc6 WatchSource:0}: Error finding container 7e41a7b51ebeb4d05f682d462ea87e6fbc744e3b050fe2d85eb4d0c959c78cc6: Status 404 returned error can't find the container with id 7e41a7b51ebeb4d05f682d462ea87e6fbc744e3b050fe2d85eb4d0c959c78cc6 Jul 08 20:00:34.813148 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.813130 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zjhks" Jul 08 20:00:34.815298 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:34.815279 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd247e9d_54a0_4db5_a1a7_404353527533.slice/crio-f5f309d781dfec46cb64e238e764d6c379e18adc12ddd79e498e95c3fb59faa0 WatchSource:0}: Error finding container f5f309d781dfec46cb64e238e764d6c379e18adc12ddd79e498e95c3fb59faa0: Status 404 returned error can't find the container with id f5f309d781dfec46cb64e238e764d6c379e18adc12ddd79e498e95c3fb59faa0 Jul 08 20:00:34.816550 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.816406 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:34.821557 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.821419 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5tkjn" Jul 08 20:00:34.825914 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:34.825874 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33019580_9b46_4bda_a0dd_3687d6e3026b.slice/crio-6f4b5401097bd703141a108d7f838884d201f58fad13e15dfb103989e054e3ef WatchSource:0}: Error finding container 6f4b5401097bd703141a108d7f838884d201f58fad13e15dfb103989e054e3ef: Status 404 returned error can't find the container with id 6f4b5401097bd703141a108d7f838884d201f58fad13e15dfb103989e054e3ef Jul 08 20:00:34.826964 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.826939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:00:34.827949 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:34.827920 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba630e65_9372_469b_8b3e_9d88d405720e.slice/crio-193c9003d611a4a0ee085b6b618d0253443402826ed995453b888e05c6188c39 WatchSource:0}: Error finding container 193c9003d611a4a0ee085b6b618d0253443402826ed995453b888e05c6188c39: Status 404 returned error can't find the container with id 193c9003d611a4a0ee085b6b618d0253443402826ed995453b888e05c6188c39 Jul 08 20:00:34.832861 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.832627 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" Jul 08 20:00:34.834127 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:34.834104 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05057eb_dc4d_45bd_adf9_1e20fdb9c027.slice/crio-7909b6e379b04cfe1a87a487e0d6ae52c9ca95b7a01e97925dfbbb8e5ac63555 WatchSource:0}: Error finding container 7909b6e379b04cfe1a87a487e0d6ae52c9ca95b7a01e97925dfbbb8e5ac63555: Status 404 returned error can't find the container with id 7909b6e379b04cfe1a87a487e0d6ae52c9ca95b7a01e97925dfbbb8e5ac63555 Jul 08 20:00:34.837751 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:34.837723 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390ebffc_4753_4b01_9f86_8116898f4c36.slice/crio-95418fdb50d11df9cabf26e933149d0d0ffe26980540dfc1a21366a3f021d803 WatchSource:0}: Error finding container 95418fdb50d11df9cabf26e933149d0d0ffe26980540dfc1a21366a3f021d803: Status 404 returned error can't find the container with id 95418fdb50d11df9cabf26e933149d0d0ffe26980540dfc1a21366a3f021d803 Jul 08 20:00:34.838629 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.838611 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-w5dc5" Jul 08 20:00:34.843927 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:34.843104 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ghlzv" Jul 08 20:00:34.853233 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:34.853210 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35b692c_4d5f_4e7a_addd_2ed017343ab4.slice/crio-8205d4730d1761a471377e1ab1e2ab99e3f9efeb40d697b35f5d282888a1125e WatchSource:0}: Error finding container 8205d4730d1761a471377e1ab1e2ab99e3f9efeb40d697b35f5d282888a1125e: Status 404 returned error can't find the container with id 8205d4730d1761a471377e1ab1e2ab99e3f9efeb40d697b35f5d282888a1125e Jul 08 20:00:34.861612 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:00:34.861595 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4364e2f1_93ad_425d_af18_8d77880860db.slice/crio-07da493d0fad66c6af4814372763d39a784bc0c85a10025c890fa08753463f54 WatchSource:0}: Error finding container 07da493d0fad66c6af4814372763d39a784bc0c85a10025c890fa08753463f54: Status 404 returned error can't find the container with id 07da493d0fad66c6af4814372763d39a784bc0c85a10025c890fa08753463f54 Jul 08 20:00:35.131453 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.131356 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:35.131453 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.131423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:35.131661 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:35.131568 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:35.131661 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:35.131626 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret podName:938c8e6d-1bc6-432e-a7fd-f1c3da1bf189 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:36.131608798 +0000 UTC m=+4.128625716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret") pod "global-pull-secret-syncer-4jt2s" (UID: "938c8e6d-1bc6-432e-a7fd-f1c3da1bf189") : object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:35.131920 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:35.131891 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:35.132043 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:35.131956 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs podName:2cb1ff8c-6dc6-4981-a487-b73961dbd578 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:36.13193818 +0000 UTC m=+4.128955112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs") pod "network-metrics-daemon-5glh6" (UID: "2cb1ff8c-6dc6-4981-a487-b73961dbd578") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:35.332380 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.332348 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:35.332567 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:35.332536 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 20:00:35.332567 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:35.332565 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 20:00:35.332681 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:35.332579 2573 projected.go:196] Error preparing data for projected volume kube-api-access-p4cz2 for pod openshift-network-diagnostics/network-check-target-8jwsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:35.332681 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:35.332645 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2 podName:ac575ae2-11c1-4daa-8d9b-4b675f53f832 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:36.332624988 +0000 UTC m=+4.329641907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4cz2" (UniqueName: "kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2") pod "network-check-target-8jwsw" (UID: "ac575ae2-11c1-4daa-8d9b-4b675f53f832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:35.568677 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.568538 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 19:55:34 +0000 UTC" deadline="2028-03-31 00:07:03.142224802 +0000 UTC" Jul 08 20:00:35.568677 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.568621 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15148h6m27.573608433s" Jul 08 20:00:35.781111 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.781075 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:35.781275 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:35.781226 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:35.789121 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.789074 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" event={"ID":"527467d3-2dd6-4220-938d-33e2b5609951","Type":"ContainerStarted","Data":"99e95d184f5c7857e22fd07e17555efbd7ded3b0a78d636f8528fdc327936741"} Jul 08 20:00:35.790531 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.790504 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerStarted","Data":"95418fdb50d11df9cabf26e933149d0d0ffe26980540dfc1a21366a3f021d803"} Jul 08 20:00:35.791813 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.791791 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-54tqn" event={"ID":"ba630e65-9372-469b-8b3e-9d88d405720e","Type":"ContainerStarted","Data":"193c9003d611a4a0ee085b6b618d0253443402826ed995453b888e05c6188c39"} Jul 08 20:00:35.793071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.793046 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" event={"ID":"09ed8740-dbe9-47b1-b6b4-d843a84df087","Type":"ContainerStarted","Data":"7e41a7b51ebeb4d05f682d462ea87e6fbc744e3b050fe2d85eb4d0c959c78cc6"} Jul 08 20:00:35.794137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.794111 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghlzv" event={"ID":"4364e2f1-93ad-425d-af18-8d77880860db","Type":"ContainerStarted","Data":"07da493d0fad66c6af4814372763d39a784bc0c85a10025c890fa08753463f54"} Jul 08 20:00:35.795442 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.795418 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w5dc5" event={"ID":"a35b692c-4d5f-4e7a-addd-2ed017343ab4","Type":"ContainerStarted","Data":"8205d4730d1761a471377e1ab1e2ab99e3f9efeb40d697b35f5d282888a1125e"} Jul 08 20:00:35.796771 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.796747 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5tkjn" event={"ID":"f05057eb-dc4d-45bd-adf9-1e20fdb9c027","Type":"ContainerStarted","Data":"7909b6e379b04cfe1a87a487e0d6ae52c9ca95b7a01e97925dfbbb8e5ac63555"} Jul 08 20:00:35.798392 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.798369 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjhks" event={"ID":"33019580-9b46-4bda-a0dd-3687d6e3026b","Type":"ContainerStarted","Data":"6f4b5401097bd703141a108d7f838884d201f58fad13e15dfb103989e054e3ef"} Jul 08 20:00:35.799526 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:35.799500 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz827" event={"ID":"bd247e9d-54a0-4db5-a1a7-404353527533","Type":"ContainerStarted","Data":"f5f309d781dfec46cb64e238e764d6c379e18adc12ddd79e498e95c3fb59faa0"} Jul 08 20:00:36.140413 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:36.140221 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:36.140413 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:36.140291 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:36.140413 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.140368 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:36.140413 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.140411 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:36.140750 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.140438 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs podName:2cb1ff8c-6dc6-4981-a487-b73961dbd578 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:38.140418554 +0000 UTC m=+6.137435471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs") pod "network-metrics-daemon-5glh6" (UID: "2cb1ff8c-6dc6-4981-a487-b73961dbd578") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:36.140750 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.140463 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret podName:938c8e6d-1bc6-432e-a7fd-f1c3da1bf189 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:38.140447341 +0000 UTC m=+6.137464257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret") pod "global-pull-secret-syncer-4jt2s" (UID: "938c8e6d-1bc6-432e-a7fd-f1c3da1bf189") : object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:36.342678 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:36.342592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:36.342830 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.342786 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 20:00:36.342830 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.342808 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 20:00:36.342830 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.342821 2573 projected.go:196] Error preparing data for projected volume kube-api-access-p4cz2 for pod openshift-network-diagnostics/network-check-target-8jwsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:36.342989 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.342878 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2 podName:ac575ae2-11c1-4daa-8d9b-4b675f53f832 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:38.342859859 +0000 UTC m=+6.339876775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4cz2" (UniqueName: "kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2") pod "network-check-target-8jwsw" (UID: "ac575ae2-11c1-4daa-8d9b-4b675f53f832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:36.569502 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:36.569435 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 19:55:34 +0000 UTC" deadline="2028-03-19 12:10:28.266560482 +0000 UTC" Jul 08 20:00:36.569502 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:36.569474 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14872h9m51.697090218s" Jul 08 20:00:36.783742 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:36.783061 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:36.783742 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.783185 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:36.783742 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:36.783561 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:36.783742 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:36.783648 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:37.780966 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:37.780931 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:37.781475 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:37.781078 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:38.156556 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:38.156475 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:38.156707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:38.156560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:38.156707 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.156701 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:38.156823 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.156765 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs podName:2cb1ff8c-6dc6-4981-a487-b73961dbd578 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:42.156742806 +0000 UTC m=+10.153759744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs") pod "network-metrics-daemon-5glh6" (UID: "2cb1ff8c-6dc6-4981-a487-b73961dbd578") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:38.157337 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.157171 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:38.157337 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.157227 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret podName:938c8e6d-1bc6-432e-a7fd-f1c3da1bf189 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:42.157211939 +0000 UTC m=+10.154228857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret") pod "global-pull-secret-syncer-4jt2s" (UID: "938c8e6d-1bc6-432e-a7fd-f1c3da1bf189") : object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:38.358526 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:38.358490 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:38.358706 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.358669 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 20:00:38.358778 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.358737 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 20:00:38.358778 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.358751 2573 projected.go:196] Error preparing data for projected volume kube-api-access-p4cz2 for pod openshift-network-diagnostics/network-check-target-8jwsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:38.358882 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.358858 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2 podName:ac575ae2-11c1-4daa-8d9b-4b675f53f832 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:42.35883782 +0000 UTC m=+10.355854734 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4cz2" (UniqueName: "kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2") pod "network-check-target-8jwsw" (UID: "ac575ae2-11c1-4daa-8d9b-4b675f53f832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:38.780686 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:38.780657 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:38.780853 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:38.780655 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:38.780853 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.780794 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:38.780960 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:38.780847 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:39.781245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:39.781164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:39.781642 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:39.781323 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:40.781404 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:40.781104 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:40.781756 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:40.781368 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:40.781756 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:40.781517 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:40.781756 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:40.781580 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:41.780778 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:41.780747 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:41.780942 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:41.780906 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:42.188018 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:42.187926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:42.188018 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:42.188010 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:42.188538 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.188092 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:42.188538 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.188107 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:42.188538 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.188175 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret podName:938c8e6d-1bc6-432e-a7fd-f1c3da1bf189 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:50.18815335 +0000 UTC m=+18.185170280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret") pod "global-pull-secret-syncer-4jt2s" (UID: "938c8e6d-1bc6-432e-a7fd-f1c3da1bf189") : object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:42.188538 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.188199 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs podName:2cb1ff8c-6dc6-4981-a487-b73961dbd578 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:50.188186481 +0000 UTC m=+18.185203397 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs") pod "network-metrics-daemon-5glh6" (UID: "2cb1ff8c-6dc6-4981-a487-b73961dbd578") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:42.389231 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:42.389173 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:42.389417 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.389352 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 20:00:42.389417 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.389375 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 20:00:42.389417 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.389388 2573 projected.go:196] Error preparing data for projected volume kube-api-access-p4cz2 for pod openshift-network-diagnostics/network-check-target-8jwsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:42.389582 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.389453 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2 podName:ac575ae2-11c1-4daa-8d9b-4b675f53f832 nodeName:}" failed. No retries permitted until 2026-07-08 20:00:50.38943498 +0000 UTC m=+18.386451910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4cz2" (UniqueName: "kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2") pod "network-check-target-8jwsw" (UID: "ac575ae2-11c1-4daa-8d9b-4b675f53f832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:42.781821 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:42.781789 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:42.781997 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.781888 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:42.781997 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:42.781950 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:42.782109 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:42.782036 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:43.781175 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:43.781141 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:43.781687 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:43.781278 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:44.780887 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:44.780845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:44.781052 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:44.780988 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:44.781135 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:44.781049 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:44.781193 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:44.781153 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:45.780449 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:45.780415 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:45.780648 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:45.780553 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:46.780511 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:46.780475 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:46.780970 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:46.780475 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:46.780970 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:46.780600 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:46.780970 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:46.780705 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:47.780336 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:47.780290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:47.780526 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:47.780459 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:48.780426 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:48.780323 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:48.780426 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:48.780359 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:48.780981 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:48.780447 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:48.780981 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:48.780534 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:48.830541 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:48.830466 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal" event={"ID":"811aeb70eccf3bdd8a847e099989ebe9","Type":"ContainerStarted","Data":"05a586bfa2a15573f7b3ae075cc5fb19dc27833be08f73f4c7c5e94757725576"} Jul 08 20:00:48.838030 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:48.837993 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" event={"ID":"527467d3-2dd6-4220-938d-33e2b5609951","Type":"ContainerStarted","Data":"31bdce0507fec959cc4f549bae077b8303973f46969bb8ae985690fb2642ff76"} Jul 08 20:00:48.847455 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:48.847397 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-18.ec2.internal" podStartSLOduration=15.847379158 podStartE2EDuration="15.847379158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:48.846510404 +0000 UTC m=+16.843527340" watchObservedRunningTime="2026-07-08 20:00:48.847379158 +0000 UTC m=+16.844396095" Jul 08 20:00:48.868244 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:48.868197 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8kdxf" podStartSLOduration=3.169687492 podStartE2EDuration="16.868182791s" totalImagesPullingTime="13.698495299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:48.867609662 +0000 UTC m=+16.864626599" watchObservedRunningTime="2026-07-08 20:00:48.868182791 +0000 UTC m=+16.865199726" Jul 08 20:00:49.781112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:49.780927 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:49.781541 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:49.781203 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:49.842915 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:49.842884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz827" event={"ID":"bd247e9d-54a0-4db5-a1a7-404353527533","Type":"ContainerStarted","Data":"fcde32e18140ce8b525942b595f1798c84b92bccd45bf36c4ff1fca7b0777e78"} Jul 08 20:00:49.869260 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:49.869190 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sz827" podStartSLOduration=3.826069344 podStartE2EDuration="17.869167906s" totalImagesPullingTime="14.043098562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:49.868062386 +0000 UTC m=+17.865079323" watchObservedRunningTime="2026-07-08 20:00:49.869167906 +0000 UTC m=+17.866184843" Jul 08 20:00:50.256027 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.255933 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:50.256172 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.256084 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:50.256172 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.256095 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:50.256172 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.256141 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret podName:938c8e6d-1bc6-432e-a7fd-f1c3da1bf189 nodeName:}" failed. No retries permitted until 2026-07-08 20:01:06.256127143 +0000 UTC m=+34.253144060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret") pod "global-pull-secret-syncer-4jt2s" (UID: "938c8e6d-1bc6-432e-a7fd-f1c3da1bf189") : object "kube-system"/"original-pull-secret" not registered Jul 08 20:00:50.256269 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.256207 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:50.256269 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.256255 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs podName:2cb1ff8c-6dc6-4981-a487-b73961dbd578 nodeName:}" failed. No retries permitted until 2026-07-08 20:01:06.256241056 +0000 UTC m=+34.253257971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs") pod "network-metrics-daemon-5glh6" (UID: "2cb1ff8c-6dc6-4981-a487-b73961dbd578") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:00:50.458250 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.458208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:50.458421 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.458349 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 20:00:50.458421 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.458366 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 20:00:50.458421 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.458375 2573 projected.go:196] Error preparing data for projected volume kube-api-access-p4cz2 for pod openshift-network-diagnostics/network-check-target-8jwsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:50.458421 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.458423 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2 podName:ac575ae2-11c1-4daa-8d9b-4b675f53f832 nodeName:}" failed. No retries permitted until 2026-07-08 20:01:06.458410168 +0000 UTC m=+34.455427082 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4cz2" (UniqueName: "kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2") pod "network-check-target-8jwsw" (UID: "ac575ae2-11c1-4daa-8d9b-4b675f53f832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:00:50.780469 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.780436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:50.780654 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.780568 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:50.780751 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.780729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:50.780885 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:50.780852 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:50.853752 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.853712 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5tkjn" event={"ID":"f05057eb-dc4d-45bd-adf9-1e20fdb9c027","Type":"ContainerStarted","Data":"05b88c169d25677db4c07cd2f8548907886737a82755b2d3b71c4afd4becfb16"} Jul 08 20:00:50.856196 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.856166 2573 generic.go:356] "Generic (PLEG): container finished" podID="33019580-9b46-4bda-a0dd-3687d6e3026b" containerID="3ddf2a5b85231fa0d4e1f1be6e3f050fa7875c588a0be268c2cfc40587a16d30" exitCode=0 Jul 08 20:00:50.856344 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.856262 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjhks" event={"ID":"33019580-9b46-4bda-a0dd-3687d6e3026b","Type":"ContainerDied","Data":"3ddf2a5b85231fa0d4e1f1be6e3f050fa7875c588a0be268c2cfc40587a16d30"} Jul 08 20:00:50.858617 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.858589 2573 generic.go:356] "Generic (PLEG): container finished" podID="9e6daa43de5249e7297b90bca40b8798" containerID="05e67a9b4b95c5d2eb1095c65aa761e5bd4d9cd11180550d528a71905efc701d" exitCode=0 Jul 08 20:00:50.858721 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.858670 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" event={"ID":"9e6daa43de5249e7297b90bca40b8798","Type":"ContainerDied","Data":"05e67a9b4b95c5d2eb1095c65aa761e5bd4d9cd11180550d528a71905efc701d"} Jul 08 20:00:50.860647 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.860531 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-54tqn" event={"ID":"ba630e65-9372-469b-8b3e-9d88d405720e","Type":"ContainerStarted","Data":"67d29122073bfedcf89a80740facd145bab01ecda143d9668f52fbbc643dca4a"} Jul 08 20:00:50.862365 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.862341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" event={"ID":"09ed8740-dbe9-47b1-b6b4-d843a84df087","Type":"ContainerStarted","Data":"222301d2c4c2ae785af1b5347bba75b10c693a6774ba421abb5fc0fc48349aa1"} Jul 08 20:00:50.864424 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.864401 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ghlzv" event={"ID":"4364e2f1-93ad-425d-af18-8d77880860db","Type":"ContainerStarted","Data":"87b982ff5d38c19b9cd88371875be3d0efa6342495f55a39d9e4b49f5ba35e62"} Jul 08 20:00:50.867167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.867147 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-w5dc5" event={"ID":"a35b692c-4d5f-4e7a-addd-2ed017343ab4","Type":"ContainerStarted","Data":"caa8d5e1217bd9c148d26c26f84df7e2f0137a985f671de9a8c06ff07670fce0"} Jul 08 20:00:50.871495 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.871446 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5tkjn" podStartSLOduration=5.157485807 podStartE2EDuration="18.871430327s" totalImagesPullingTime="13.71394452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:50.87112415 +0000 UTC m=+18.868141090" watchObservedRunningTime="2026-07-08 20:00:50.871430327 +0000 UTC m=+18.868447264" Jul 08 20:00:50.930944 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.930885 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-54tqn" podStartSLOduration=5.211269961 podStartE2EDuration="18.930863435s" totalImagesPullingTime="13.719593474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:50.930044204 +0000 UTC m=+18.927061142" watchObservedRunningTime="2026-07-08 20:00:50.930863435 +0000 UTC m=+18.927880373" Jul 08 20:00:50.946323 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.946252 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-w5dc5" podStartSLOduration=8.238392023 podStartE2EDuration="17.946234988s" totalImagesPullingTime="9.707842965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:50.946212526 +0000 UTC m=+18.943229461" watchObservedRunningTime="2026-07-08 20:00:50.946234988 +0000 UTC m=+18.943251926" Jul 08 20:00:50.962343 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.961719 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ghlzv" podStartSLOduration=4.279488932 podStartE2EDuration="17.961700406s" totalImagesPullingTime="13.682211474s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:50.961456389 +0000 UTC m=+18.958473324" watchObservedRunningTime="2026-07-08 20:00:50.961700406 +0000 UTC m=+18.958717343" Jul 08 20:00:50.965174 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:50.965147 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:51.083702 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.083617 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:51.150910 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.150877 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:51.231796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.231768 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:51.371112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.370951 2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 20:00:51.640640 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.640447 2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T20:00:51.371106343Z","UUID":"bf3ac2ce-3ca3-473f-85a2-45effaea3afc","Handler":null,"Name":"","Endpoint":""} Jul 08 20:00:51.642269 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.642227 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 20:00:51.642417 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.642276 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 20:00:51.780797 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.780765 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:51.780965 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:51.780889 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:51.870629 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.870587 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" event={"ID":"9e6daa43de5249e7297b90bca40b8798","Type":"ContainerStarted","Data":"4fb758a10e9453383c815e0f5059719cde7601ae9cbe91472bba74b24269fdf3"} Jul 08 20:00:51.873140 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.873074 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" event={"ID":"09ed8740-dbe9-47b1-b6b4-d843a84df087","Type":"ContainerStarted","Data":"314264e2eed9143d458bd32491c50aab73ac1e9cf7eeecc564f85e66781ecdb6"} Jul 08 20:00:51.888586 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:51.888532 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-18.ec2.internal" podStartSLOduration=18.888514843 podStartE2EDuration="18.888514843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:51.888064114 +0000 UTC m=+19.885081051" watchObservedRunningTime="2026-07-08 20:00:51.888514843 +0000 UTC m=+19.885531779" Jul 08 20:00:52.450632 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:52.450587 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:52.533376 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:52.533328 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:52.781942 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:52.781859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:52.782109 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:52.781984 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:52.782227 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:52.782212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:52.782357 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:52.782332 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:53.780518 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:53.780480 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:53.780914 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:53.780611 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:54.613491 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:54.613459 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:54.714252 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:54.714209 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:54.780498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:54.780467 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:54.780657 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:54.780593 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:54.780994 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:54.780671 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:54.780994 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:54.780779 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:55.363387 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:55.363358 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:55.363976 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:55.363953 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:55.781081 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:55.781013 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:55.781516 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:55.781136 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:55.881703 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:55.881676 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:55.882116 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:55.882094 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-54tqn" Jul 08 20:00:55.951429 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:55.951395 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:56.780631 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:56.780594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:56.780783 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:56.780594 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:56.780783 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:56.780727 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:56.781584 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:56.781547 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:57.322443 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:57.322412 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:57.780335 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:57.780256 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:57.780484 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:57.780405 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:57.925476 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:57.925440 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:58.492709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:58.492674 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:58.591833 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:58.591805 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:00:58.780440 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:58.780419 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:00:58.780523 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:58.780511 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:00:58.780627 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:58.780610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:00:58.780726 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:58.780708 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:00:59.781132 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.780967 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:00:59.781673 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:00:59.781209 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:00:59.891343 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.891274 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" event={"ID":"09ed8740-dbe9-47b1-b6b4-d843a84df087","Type":"ContainerStarted","Data":"b150405e78128d4d35da6ca35879fa795ab777e0442a5472e6857a3d027a930f"} Jul 08 20:00:59.893160 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.893136 2573 generic.go:356] "Generic (PLEG): container finished" podID="33019580-9b46-4bda-a0dd-3687d6e3026b" containerID="376b9f5e24644eda8b803a083c7127ba3276032effe2a666748159179bd0c895" exitCode=0 Jul 08 20:00:59.893287 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.893215 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjhks" event={"ID":"33019580-9b46-4bda-a0dd-3687d6e3026b","Type":"ContainerDied","Data":"376b9f5e24644eda8b803a083c7127ba3276032effe2a666748159179bd0c895"} Jul 08 20:00:59.895950 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.895932 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:00:59.896282 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.896263 2573 generic.go:356] "Generic (PLEG): container finished" podID="390ebffc-4753-4b01-9f86-8116898f4c36" containerID="ba6a11dee0b888402c8ef8254082565812516261fce51af6331a8133d575b97c" exitCode=1 Jul 08 20:00:59.896399 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.896295 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerStarted","Data":"bf4e295c702b4c28bd26928d651f55416f6bd217d91dfd5c6f77921a9c268d4f"} Jul 08 20:00:59.896399 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.896374 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerStarted","Data":"cf49c8c1ac592cb56940ec5ba97f676da847aba2a18b65bf5bb452abcb5f2c75"} Jul 08 20:00:59.896506 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.896426 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerStarted","Data":"5a5cd7cc88b94c63354ef6ed5fb6faa72c133a88761590e7228af43a49810781"} Jul 08 20:00:59.896561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.896509 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerStarted","Data":"6f87ca45a7cf5e8e9968442769c8c3734fe60fe9c45d943b4fc3756951840032"} Jul 08 20:00:59.896613 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.896559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerDied","Data":"ba6a11dee0b888402c8ef8254082565812516261fce51af6331a8133d575b97c"} Jul 08 20:00:59.896666 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.896610 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerStarted","Data":"b892527d8ee8ad2707a83b1660ca7b5700ba0ef9a67358e5c47584cbf4f2d58a"} Jul 08 20:00:59.920662 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.920622 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9dt8h" podStartSLOduration=5.253320731 podStartE2EDuration="27.920611501s" totalImagesPullingTime="22.66729077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:00:59.9205324 +0000 UTC m=+27.917549336" watchObservedRunningTime="2026-07-08 20:00:59.920611501 +0000 UTC m=+27.917628436" Jul 08 20:00:59.993773 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:00:59.993750 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:00.071401 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:00.071295 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:00.454529 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:00.454449 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:00.487851 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:00.487826 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w5dc5_a35b692c-4d5f-4e7a-addd-2ed017343ab4/dns-node-resolver/0.log" Jul 08 20:01:00.571156 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:00.571129 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:00.781203 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:00.781121 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:00.781203 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:00.781160 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:00.781747 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:00.781254 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:00.781747 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:00.781374 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:00.887374 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:00.887346 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5tkjn_f05057eb-dc4d-45bd-adf9-1e20fdb9c027/node-ca/0.log" Jul 08 20:01:01.013624 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:01.013592 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:01.780868 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:01.780777 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:01.781033 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:01.780894 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:01.904924 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:01.904892 2573 generic.go:356] "Generic (PLEG): container finished" podID="33019580-9b46-4bda-a0dd-3687d6e3026b" containerID="135d5b8508ee721fdded49ee2f17c4f974dbc064b2ed74fc30f18527fdf50ad4" exitCode=0 Jul 08 20:01:01.905401 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:01.904982 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjhks" event={"ID":"33019580-9b46-4bda-a0dd-3687d6e3026b","Type":"ContainerDied","Data":"135d5b8508ee721fdded49ee2f17c4f974dbc064b2ed74fc30f18527fdf50ad4"} Jul 08 20:01:01.908062 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:01.908043 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:01:01.908390 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:01.908371 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerStarted","Data":"0591b6cb093a3baccf1ecdf42e8a2c4d1e1bd91657b494ac1484391cbe63c3b0"} Jul 08 20:01:02.781839 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:02.781806 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:02.782014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:02.781878 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:02.782014 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:02.781894 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:02.782014 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:02.781960 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:03.781049 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:03.780867 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:03.781523 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:03.781126 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:03.915610 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:03.915578 2573 generic.go:356] "Generic (PLEG): container finished" podID="33019580-9b46-4bda-a0dd-3687d6e3026b" containerID="005003010cb06c21a04c1457b997968bc06084204b155d53eee134e6835df1ca" exitCode=0 Jul 08 20:01:03.915772 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:03.915658 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjhks" event={"ID":"33019580-9b46-4bda-a0dd-3687d6e3026b","Type":"ContainerDied","Data":"005003010cb06c21a04c1457b997968bc06084204b155d53eee134e6835df1ca"} Jul 08 20:01:03.919047 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:03.919028 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:01:03.919409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:03.919387 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerStarted","Data":"b2f1cf921203287deb7760b8c145cedaf55be14ea04d26fb75a55c19783c7eec"} Jul 08 20:01:03.919675 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:03.919654 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:01:03.919830 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:03.919814 2573 scope.go:117] "RemoveContainer" containerID="ba6a11dee0b888402c8ef8254082565812516261fce51af6331a8133d575b97c" Jul 08 20:01:03.937655 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:03.937629 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:01:04.780679 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:04.780644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:04.780833 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:04.780650 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:04.780833 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:04.780761 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:04.780833 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:04.780819 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:04.926560 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:04.926532 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:01:04.927055 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:04.927024 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" event={"ID":"390ebffc-4753-4b01-9f86-8116898f4c36","Type":"ContainerStarted","Data":"d4be61e4449edffd54c71ed5470b6ce50715abd9adfc71df362a2dd0cc05c83a"} Jul 08 20:01:04.927332 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:04.927268 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:01:04.927424 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:04.927404 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:01:04.944755 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:04.944725 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:01:04.996536 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:04.996475 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" podStartSLOduration=8.891037054 podStartE2EDuration="32.996459925s" totalImagesPullingTime="24.105422871s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:04.963895665 +0000 UTC m=+32.960912602" watchObservedRunningTime="2026-07-08 20:01:04.996459925 +0000 UTC m=+32.993476889" Jul 08 20:01:05.780730 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:05.780703 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:05.780877 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:05.780839 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:06.004722 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:06.004694 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:06.211858 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:06.211824 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:06.283699 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:06.283673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:06.283844 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:06.283713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:06.283844 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.283816 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:01:06.283906 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.283862 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 20:01:06.283906 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.283886 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs podName:2cb1ff8c-6dc6-4981-a487-b73961dbd578 nodeName:}" failed. No retries permitted until 2026-07-08 20:01:38.283864647 +0000 UTC m=+66.280881583 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs") pod "network-metrics-daemon-5glh6" (UID: "2cb1ff8c-6dc6-4981-a487-b73961dbd578") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 20:01:06.283906 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.283904 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret podName:938c8e6d-1bc6-432e-a7fd-f1c3da1bf189 nodeName:}" failed. No retries permitted until 2026-07-08 20:01:38.283895286 +0000 UTC m=+66.280912201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret") pod "global-pull-secret-syncer-4jt2s" (UID: "938c8e6d-1bc6-432e-a7fd-f1c3da1bf189") : object "kube-system"/"original-pull-secret" not registered Jul 08 20:01:06.341401 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:06.341373 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:06.484890 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:06.484812 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:06.485017 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.484999 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 20:01:06.485068 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.485022 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 20:01:06.485068 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.485035 2573 projected.go:196] Error preparing data for projected volume kube-api-access-p4cz2 for pod openshift-network-diagnostics/network-check-target-8jwsw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:01:06.485162 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.485095 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2 podName:ac575ae2-11c1-4daa-8d9b-4b675f53f832 nodeName:}" failed. No retries permitted until 2026-07-08 20:01:38.485077764 +0000 UTC m=+66.482094678 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-p4cz2" (UniqueName: "kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2") pod "network-check-target-8jwsw" (UID: "ac575ae2-11c1-4daa-8d9b-4b675f53f832") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 20:01:06.780651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:06.780576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:06.780801 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:06.780589 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:06.780801 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.780716 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:06.780917 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:06.780803 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:07.041801 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:07.041727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:07.138970 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:07.138931 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:07.780304 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:07.780270 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:07.780472 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:07.780431 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:07.895506 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:07.895472 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:08.781032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:08.780982 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:08.781032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:08.781019 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:08.781541 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:08.781125 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:08.781541 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:08.781274 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:08.915871 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:08.915836 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:09.035243 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:09.035153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:09.780896 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:09.780866 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:09.781060 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:09.780982 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:09.919047 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:09.919009 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:09.940161 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:09.940133 2573 generic.go:356] "Generic (PLEG): container finished" podID="33019580-9b46-4bda-a0dd-3687d6e3026b" containerID="4203d9e1a007d417519e8bdd98abfeddab552610a6e73ff73e44ad94ff370c7f" exitCode=0 Jul 08 20:01:09.940267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:09.940173 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjhks" event={"ID":"33019580-9b46-4bda-a0dd-3687d6e3026b","Type":"ContainerDied","Data":"4203d9e1a007d417519e8bdd98abfeddab552610a6e73ff73e44ad94ff370c7f"} Jul 08 20:01:09.999108 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:09.999071 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:10.113525 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:10.113490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:10.459655 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:10.459623 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:10.527372 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:10.527343 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:10.751513 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:10.751443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:10.780665 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:10.780633 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:10.780809 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:10.780766 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:10.780860 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:10.780810 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:10.780926 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:10.780900 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:10.928953 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:10.928910 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:10.945568 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:10.945536 2573 generic.go:356] "Generic (PLEG): container finished" podID="33019580-9b46-4bda-a0dd-3687d6e3026b" containerID="d1d1af1f0ab13dcc40455108cd7bf01e4d3afabb3f738a0c3d01720f34b87362" exitCode=0 Jul 08 20:01:10.945696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:10.945581 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjhks" event={"ID":"33019580-9b46-4bda-a0dd-3687d6e3026b","Type":"ContainerDied","Data":"d1d1af1f0ab13dcc40455108cd7bf01e4d3afabb3f738a0c3d01720f34b87362"} Jul 08 20:01:11.302976 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:11.302889 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:11.697725 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:11.697697 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:11.780995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:11.780968 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:11.781152 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:11.781107 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:11.951320 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:11.951228 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zjhks" event={"ID":"33019580-9b46-4bda-a0dd-3687d6e3026b","Type":"ContainerStarted","Data":"d665ba736463f46221221a68ece496cd12d401af9749e60359045e1391535c20"} Jul 08 20:01:11.983210 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:11.983155 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zjhks" podStartSLOduration=13.165891567 podStartE2EDuration="39.983141739s" totalImagesPullingTime="26.817250172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:11.982659983 +0000 UTC m=+39.979676919" watchObservedRunningTime="2026-07-08 20:01:11.983141739 +0000 UTC m=+39.980158675" Jul 08 20:01:12.114206 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:12.114170 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:12.191115 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:12.191086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:12.518691 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:12.518661 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:12.587350 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:12.587299 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:12.704701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:12.704516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:12.781201 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:12.781123 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:12.781201 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:12.781192 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:12.781403 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:12.781211 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:12.781403 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:12.781266 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:12.839180 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:12.839139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:13.300892 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:13.300863 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:13.781129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:13.781098 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:13.781301 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:13.781207 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:13.900524 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:13.900491 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:14.096475 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:14.096395 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:14.309709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:14.309666 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:14.727051 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:14.727013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:14.781159 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:14.781133 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:14.781301 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:14.781143 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:14.781372 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:14.781291 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:14.781372 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:14.781302 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:15.305330 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:15.305288 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:15.702929 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:15.702903 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:15.781257 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:15.781227 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:15.781450 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:15.781370 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:16.101179 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:16.101108 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:16.307487 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:16.307448 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:16.499865 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:16.499838 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:16.780565 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:16.780472 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:16.780993 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:16.780570 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:16.780993 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:16.780653 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:16.780993 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:16.780759 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:16.900396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:16.900358 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:17.780954 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:17.780909 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:17.781347 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:17.781032 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:18.239254 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:18.239224 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:18.780545 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:18.780514 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:18.780740 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:18.780549 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:18.780740 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:18.780661 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:18.780740 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:18.780697 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:19.780638 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:19.780610 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:19.781004 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:19.780723 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:20.781193 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:20.781159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:20.781602 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:20.781274 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:20.781602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:20.781327 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:20.781602 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:20.781413 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:21.781239 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:21.781196 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:21.781971 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:21.781939 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:21.841003 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:21.840956 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:22.030507 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.030448 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:22.147909 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.147806 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:22.687195 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.687146 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:22.771587 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.771556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:22.781655 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.781630 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:22.782024 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:22.781709 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:22.782024 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.781816 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:22.782024 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:22.781928 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:22.958347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.958272 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:22.988334 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.987965 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4jt2s"] Jul 08 20:01:22.988334 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.988086 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:22.988334 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:22.988189 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:22.988832 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.988807 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5glh6"] Jul 08 20:01:22.988944 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.988917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:22.989034 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:22.989015 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:22.989609 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.989591 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jwsw"] Jul 08 20:01:22.989678 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:22.989669 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:22.989775 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:22.989756 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:23.165047 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:23.165020 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:23.370606 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:23.370536 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:24.130981 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:24.130902 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:24.541526 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:24.541503 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:24.650921 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:24.650892 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:24.753200 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:24.753166 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:24.781282 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:24.781228 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:24.781282 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:24.781236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:24.781282 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:24.781249 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:24.781503 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:24.781361 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8jwsw" podUID="ac575ae2-11c1-4daa-8d9b-4b675f53f832" Jul 08 20:01:24.781503 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:24.781466 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4jt2s" podUID="938c8e6d-1bc6-432e-a7fd-f1c3da1bf189" Jul 08 20:01:24.781608 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:24.781530 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5glh6" podUID="2cb1ff8c-6dc6-4981-a487-b73961dbd578" Jul 08 20:01:25.153768 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:25.153687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:25.271845 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:25.271817 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:25.372918 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:25.372880 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:26.711490 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.711462 2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-18.ec2.internal" event="NodeReady" Jul 08 20:01:26.711862 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.711579 2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 20:01:26.768444 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.768417 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v7nfw"] Jul 08 20:01:26.770069 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.770054 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.774112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.774094 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-psksh\"" Jul 08 20:01:26.774729 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.774708 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 20:01:26.774825 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.774744 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 20:01:26.780543 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.780525 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:26.780638 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.780552 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:26.780780 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.780761 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:26.783783 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.783767 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cwvxn"] Jul 08 20:01:26.784740 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.784726 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zs499\"" Jul 08 20:01:26.785254 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.785237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cwvxn" Jul 08 20:01:26.785427 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.785394 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 20:01:26.785523 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.785502 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqvk8\"" Jul 08 20:01:26.785852 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.785837 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 20:01:26.785957 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.785874 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 20:01:26.785957 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.785919 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 20:01:26.792277 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.792255 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 20:01:26.792450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.792384 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 20:01:26.792840 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.792813 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 20:01:26.792965 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.792950 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-6mhjs\"" Jul 08 20:01:26.797986 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.797965 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v7nfw"] Jul 08 20:01:26.808585 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.808564 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cwvxn"] Jul 08 20:01:26.819565 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.819548 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/611e0c1c-93b6-4a90-8165-7128c8ff6be9-cert\") pod \"ingress-canary-cwvxn\" (UID: \"611e0c1c-93b6-4a90-8165-7128c8ff6be9\") " pod="openshift-ingress-canary/ingress-canary-cwvxn" Jul 08 20:01:26.819654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.819571 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7fhp\" (UniqueName: \"kubernetes.io/projected/611e0c1c-93b6-4a90-8165-7128c8ff6be9-kube-api-access-t7fhp\") pod \"ingress-canary-cwvxn\" (UID: \"611e0c1c-93b6-4a90-8165-7128c8ff6be9\") " pod="openshift-ingress-canary/ingress-canary-cwvxn" Jul 08 20:01:26.819654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.819590 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4cb57bc-18c6-4bde-846c-d3a761bb4848-config-volume\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.819654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.819647 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4cb57bc-18c6-4bde-846c-d3a761bb4848-metrics-tls\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.819822 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.819736 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57849\" (UniqueName: \"kubernetes.io/projected/f4cb57bc-18c6-4bde-846c-d3a761bb4848-kube-api-access-57849\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.819822 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.819788 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cb57bc-18c6-4bde-846c-d3a761bb4848-tmp-dir\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.885190 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.885151 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lmwhx"] Jul 08 20:01:26.887058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.887039 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:26.890922 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.890896 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 20:01:26.891148 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.891134 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-48jvt\"" Jul 08 20:01:26.891409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.891394 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 20:01:26.892039 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.892019 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 20:01:26.892131 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.892079 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 20:01:26.912006 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.911980 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lmwhx"] Jul 08 20:01:26.920563 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920544 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cb57bc-18c6-4bde-846c-d3a761bb4848-tmp-dir\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.920666 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/611e0c1c-93b6-4a90-8165-7128c8ff6be9-cert\") pod \"ingress-canary-cwvxn\" (UID: \"611e0c1c-93b6-4a90-8165-7128c8ff6be9\") " pod="openshift-ingress-canary/ingress-canary-cwvxn" Jul 08 20:01:26.920666 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920598 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7fhp\" (UniqueName: \"kubernetes.io/projected/611e0c1c-93b6-4a90-8165-7128c8ff6be9-kube-api-access-t7fhp\") pod \"ingress-canary-cwvxn\" (UID: \"611e0c1c-93b6-4a90-8165-7128c8ff6be9\") " pod="openshift-ingress-canary/ingress-canary-cwvxn" Jul 08 20:01:26.920666 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920617 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4cb57bc-18c6-4bde-846c-d3a761bb4848-config-volume\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.920666 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920642 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4cb57bc-18c6-4bde-846c-d3a761bb4848-metrics-tls\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.920825 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d128df57-bf08-4aeb-9513-d64f27bcf180-data-volume\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:26.920825 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920700 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d128df57-bf08-4aeb-9513-d64f27bcf180-crio-socket\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:26.920825 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920717 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d128df57-bf08-4aeb-9513-d64f27bcf180-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:26.920825 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57849\" (UniqueName: \"kubernetes.io/projected/f4cb57bc-18c6-4bde-846c-d3a761bb4848-kube-api-access-57849\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.920825 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920780 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d128df57-bf08-4aeb-9513-d64f27bcf180-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:26.920825 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddw7v\" (UniqueName: \"kubernetes.io/projected/d128df57-bf08-4aeb-9513-d64f27bcf180-kube-api-access-ddw7v\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:26.921035 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.920873 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4cb57bc-18c6-4bde-846c-d3a761bb4848-tmp-dir\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.921362 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.921344 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f4cb57bc-18c6-4bde-846c-d3a761bb4848-config-volume\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.924292 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.924272 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4cb57bc-18c6-4bde-846c-d3a761bb4848-metrics-tls\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:26.924585 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.924565 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/611e0c1c-93b6-4a90-8165-7128c8ff6be9-cert\") pod \"ingress-canary-cwvxn\" (UID: \"611e0c1c-93b6-4a90-8165-7128c8ff6be9\") " pod="openshift-ingress-canary/ingress-canary-cwvxn" Jul 08 20:01:26.929785 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.929755 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7fhp\" (UniqueName: \"kubernetes.io/projected/611e0c1c-93b6-4a90-8165-7128c8ff6be9-kube-api-access-t7fhp\") pod \"ingress-canary-cwvxn\" (UID: \"611e0c1c-93b6-4a90-8165-7128c8ff6be9\") " pod="openshift-ingress-canary/ingress-canary-cwvxn" Jul 08 20:01:26.929867 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:26.929807 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57849\" (UniqueName: \"kubernetes.io/projected/f4cb57bc-18c6-4bde-846c-d3a761bb4848-kube-api-access-57849\") pod \"dns-default-v7nfw\" (UID: \"f4cb57bc-18c6-4bde-846c-d3a761bb4848\") " pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:27.021808 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.021786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d128df57-bf08-4aeb-9513-d64f27bcf180-crio-socket\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.021808 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.021810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d128df57-bf08-4aeb-9513-d64f27bcf180-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.021944 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.021835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d128df57-bf08-4aeb-9513-d64f27bcf180-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.021944 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.021852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddw7v\" (UniqueName: \"kubernetes.io/projected/d128df57-bf08-4aeb-9513-d64f27bcf180-kube-api-access-ddw7v\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.021944 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.021898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d128df57-bf08-4aeb-9513-d64f27bcf180-data-volume\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.022044 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.021984 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d128df57-bf08-4aeb-9513-d64f27bcf180-crio-socket\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.022207 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.022184 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d128df57-bf08-4aeb-9513-d64f27bcf180-data-volume\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.022396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.022376 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d128df57-bf08-4aeb-9513-d64f27bcf180-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.023812 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.023796 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d128df57-bf08-4aeb-9513-d64f27bcf180-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.030684 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.030662 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddw7v\" (UniqueName: \"kubernetes.io/projected/d128df57-bf08-4aeb-9513-d64f27bcf180-kube-api-access-ddw7v\") pod \"insights-runtime-extractor-lmwhx\" (UID: \"d128df57-bf08-4aeb-9513-d64f27bcf180\") " pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.038114 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.038039 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:27.079107 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.079057 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:27.107217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.107189 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cwvxn" Jul 08 20:01:27.195715 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.195686 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lmwhx" Jul 08 20:01:27.237269 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.237239 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v7nfw"] Jul 08 20:01:27.240764 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:27.240736 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cb57bc_18c6_4bde_846c_d3a761bb4848.slice/crio-a72d004bc193a571c2f142c07828a7e7ce100333dc5f94633e479ce507376692 WatchSource:0}: Error finding container a72d004bc193a571c2f142c07828a7e7ce100333dc5f94633e479ce507376692: Status 404 returned error can't find the container with id a72d004bc193a571c2f142c07828a7e7ce100333dc5f94633e479ce507376692 Jul 08 20:01:27.253896 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.253829 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cwvxn"] Jul 08 20:01:27.257152 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:27.257123 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611e0c1c_93b6_4a90_8165_7128c8ff6be9.slice/crio-3d41ec6ba8b5dd6388e2aa8bd9aa2f52a96b2ab3781e3fb157afefa518fdc6f3 WatchSource:0}: Error finding container 3d41ec6ba8b5dd6388e2aa8bd9aa2f52a96b2ab3781e3fb157afefa518fdc6f3: Status 404 returned error can't find the container with id 3d41ec6ba8b5dd6388e2aa8bd9aa2f52a96b2ab3781e3fb157afefa518fdc6f3 Jul 08 20:01:27.324917 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.324830 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lmwhx"] Jul 08 20:01:27.327819 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:27.327797 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd128df57_bf08_4aeb_9513_d64f27bcf180.slice/crio-8cae0b5647006e1c9ca541c9107f1c2b8083d25200e9b2722c3c8c7b4e64349a WatchSource:0}: Error finding container 8cae0b5647006e1c9ca541c9107f1c2b8083d25200e9b2722c3c8c7b4e64349a: Status 404 returned error can't find the container with id 8cae0b5647006e1c9ca541c9107f1c2b8083d25200e9b2722c3c8c7b4e64349a Jul 08 20:01:27.700147 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.700109 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:27.793490 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.793454 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:27.988242 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.988161 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmwhx" event={"ID":"d128df57-bf08-4aeb-9513-d64f27bcf180","Type":"ContainerStarted","Data":"8cae0b5647006e1c9ca541c9107f1c2b8083d25200e9b2722c3c8c7b4e64349a"} Jul 08 20:01:27.990066 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.989944 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cwvxn" event={"ID":"611e0c1c-93b6-4a90-8165-7128c8ff6be9","Type":"ContainerStarted","Data":"3d41ec6ba8b5dd6388e2aa8bd9aa2f52a96b2ab3781e3fb157afefa518fdc6f3"} Jul 08 20:01:27.991974 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:27.991832 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7nfw" event={"ID":"f4cb57bc-18c6-4bde-846c-d3a761bb4848","Type":"ContainerStarted","Data":"a72d004bc193a571c2f142c07828a7e7ce100333dc5f94633e479ce507376692"} Jul 08 20:01:28.740994 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:28.740958 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:28.834189 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:28.834151 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:28.907535 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:28.907497 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:28.910370 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:28.910034 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq"] Jul 08 20:01:28.924440 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:28.924079 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq"] Jul 08 20:01:28.924440 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:28.924196 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" Jul 08 20:01:28.927352 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:28.926964 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 08 20:01:28.927352 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:28.927193 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2lhft\"" Jul 08 20:01:29.037757 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:29.037678 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3fcb04b1-40e5-401d-aecf-8304c3b4933f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-ffngq\" (UID: \"3fcb04b1-40e5-401d-aecf-8304c3b4933f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" Jul 08 20:01:29.138469 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:29.138433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3fcb04b1-40e5-401d-aecf-8304c3b4933f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-ffngq\" (UID: \"3fcb04b1-40e5-401d-aecf-8304c3b4933f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" Jul 08 20:01:29.138616 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:29.138566 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 08 20:01:29.138683 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:29.138623 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fcb04b1-40e5-401d-aecf-8304c3b4933f-tls-certificates podName:3fcb04b1-40e5-401d-aecf-8304c3b4933f nodeName:}" failed. No retries permitted until 2026-07-08 20:01:29.638608362 +0000 UTC m=+57.635625275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/3fcb04b1-40e5-401d-aecf-8304c3b4933f-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-ffngq" (UID: "3fcb04b1-40e5-401d-aecf-8304c3b4933f") : secret "prometheus-operator-admission-webhook-tls" not found Jul 08 20:01:29.200726 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:29.200693 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:29.642568 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:29.642533 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3fcb04b1-40e5-401d-aecf-8304c3b4933f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-ffngq\" (UID: \"3fcb04b1-40e5-401d-aecf-8304c3b4933f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" Jul 08 20:01:29.645159 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:29.645135 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3fcb04b1-40e5-401d-aecf-8304c3b4933f-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-ffngq\" (UID: \"3fcb04b1-40e5-401d-aecf-8304c3b4933f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" Jul 08 20:01:29.838281 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:29.838237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" Jul 08 20:01:31.166365 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:31.166335 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:31.235608 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:31.235581 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:32.383846 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.383812 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9454698c-8cf8b"] Jul 08 20:01:32.391194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.391173 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.398787 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.398764 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 20:01:32.398898 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.398802 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 20:01:32.398898 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.398763 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 20:01:32.399354 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.399338 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 20:01:32.399959 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.399937 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 20:01:32.400061 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.400002 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 20:01:32.400204 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.400179 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 20:01:32.400204 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.400196 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xklpj\"" Jul 08 20:01:32.411596 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.411578 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9454698c-8cf8b"] Jul 08 20:01:32.465044 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.465020 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-oauth-serving-cert\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.465179 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.465047 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4msx4\" (UniqueName: \"kubernetes.io/projected/a61507dc-6151-4502-8b12-5d1604a83316-kube-api-access-4msx4\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.465179 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.465091 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-serving-cert\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.465296 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.465188 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-oauth-config\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.465296 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.465231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-service-ca\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.465296 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.465280 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-console-config\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.565858 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.565831 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4msx4\" (UniqueName: \"kubernetes.io/projected/a61507dc-6151-4502-8b12-5d1604a83316-kube-api-access-4msx4\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.566007 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.565985 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-serving-cert\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.566123 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.566049 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-oauth-config\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.566123 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.566073 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-service-ca\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.566219 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.566161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-console-config\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.566258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.566215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-oauth-serving-cert\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.570551 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.570525 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 08 20:01:32.570681 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.570659 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 08 20:01:32.570808 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.570659 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 08 20:01:32.570808 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.570744 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 08 20:01:32.570808 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.570778 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 08 20:01:32.576862 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.576843 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-service-ca\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.576862 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.576850 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-console-config\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.577017 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.576944 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-oauth-serving-cert\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.578357 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.578338 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 20:01:32.578758 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.578738 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-oauth-config\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.578868 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.578848 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-serving-cert\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.589571 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.589554 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 20:01:32.598841 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.598825 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4msx4\" (UniqueName: \"kubernetes.io/projected/a61507dc-6151-4502-8b12-5d1604a83316-kube-api-access-4msx4\") pod \"console-9454698c-8cf8b\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:32.704292 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.704269 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xklpj\"" Jul 08 20:01:32.711773 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:32.711742 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:33.012377 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:33.012345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7nfw" event={"ID":"f4cb57bc-18c6-4bde-846c-d3a761bb4848","Type":"ContainerStarted","Data":"93e9c3328be68a61ba87a796c066c0e534b4bbae081ad56b165cea8ea79cae35"} Jul 08 20:01:33.014156 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:33.014133 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq"] Jul 08 20:01:33.015489 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:33.015415 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmwhx" event={"ID":"d128df57-bf08-4aeb-9513-d64f27bcf180","Type":"ContainerStarted","Data":"ef32de64ec8beb516a65976ef0d0bfef2500d537411537f580c26b8f64daa969"} Jul 08 20:01:33.019642 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:33.019611 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fcb04b1_40e5_401d_aecf_8304c3b4933f.slice/crio-1ea26e4b1c6963e88e2f48afd610cbf156987ab58852fcbd1e41c1088cb42ed2 WatchSource:0}: Error finding container 1ea26e4b1c6963e88e2f48afd610cbf156987ab58852fcbd1e41c1088cb42ed2: Status 404 returned error can't find the container with id 1ea26e4b1c6963e88e2f48afd610cbf156987ab58852fcbd1e41c1088cb42ed2 Jul 08 20:01:33.035204 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:33.035041 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9454698c-8cf8b"] Jul 08 20:01:33.037701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:33.037594 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cwvxn" podStartSLOduration=1.404076406 podStartE2EDuration="7.037574824s" totalImagesPullingTime="5.633498418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:33.03692878 +0000 UTC m=+61.033945716" watchObservedRunningTime="2026-07-08 20:01:33.037574824 +0000 UTC m=+61.034591761" Jul 08 20:01:33.043377 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:33.043338 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61507dc_6151_4502_8b12_5d1604a83316.slice/crio-9a485228ae7d553554f7edf8ba81101a0eb13260709dc3b52d968b1df8e15ef7 WatchSource:0}: Error finding container 9a485228ae7d553554f7edf8ba81101a0eb13260709dc3b52d968b1df8e15ef7: Status 404 returned error can't find the container with id 9a485228ae7d553554f7edf8ba81101a0eb13260709dc3b52d968b1df8e15ef7 Jul 08 20:01:33.374671 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:33.374594 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:33.472701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:33.472663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:34.021946 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:34.021908 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" event={"ID":"3fcb04b1-40e5-401d-aecf-8304c3b4933f","Type":"ContainerStarted","Data":"1ea26e4b1c6963e88e2f48afd610cbf156987ab58852fcbd1e41c1088cb42ed2"} Jul 08 20:01:34.024583 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:34.024478 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v7nfw" event={"ID":"f4cb57bc-18c6-4bde-846c-d3a761bb4848","Type":"ContainerStarted","Data":"d04f44a0a85100042c0af2d328203a931854f97d301f0b8adc19940c1bca01e3"} Jul 08 20:01:34.024583 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:34.024538 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:34.026098 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:34.026077 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9454698c-8cf8b" event={"ID":"a61507dc-6151-4502-8b12-5d1604a83316","Type":"ContainerStarted","Data":"9a485228ae7d553554f7edf8ba81101a0eb13260709dc3b52d968b1df8e15ef7"} Jul 08 20:01:34.028322 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:34.028280 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmwhx" event={"ID":"d128df57-bf08-4aeb-9513-d64f27bcf180","Type":"ContainerStarted","Data":"a9c8346056b74322e98c9c3b1667a56ae44afae7d177f7f98d7301df475ba530"} Jul 08 20:01:34.029956 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:34.029932 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cwvxn" event={"ID":"611e0c1c-93b6-4a90-8165-7128c8ff6be9","Type":"ContainerStarted","Data":"019e2e82d4048e1f9fd1c92e8217e3a1fb54f08b392d4663c120d604016a23aa"} Jul 08 20:01:34.042781 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:34.042667 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v7nfw" podStartSLOduration=2.421625912 podStartE2EDuration="8.042649796s" totalImagesPullingTime="5.621023884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:34.042545416 +0000 UTC m=+62.039562349" watchObservedRunningTime="2026-07-08 20:01:34.042649796 +0000 UTC m=+62.039666734" Jul 08 20:01:35.034626 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:35.034561 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" event={"ID":"3fcb04b1-40e5-401d-aecf-8304c3b4933f","Type":"ContainerStarted","Data":"df3d9f3e577d43f86684ecff323132b41bbfe5db7989f933425469016339f88b"} Jul 08 20:01:35.035080 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:35.035044 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" Jul 08 20:01:35.043735 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:35.041019 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" Jul 08 20:01:35.052041 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:35.051985 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-ffngq" podStartSLOduration=5.57556095 podStartE2EDuration="7.051966385s" totalImagesPullingTime="1.476405435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:35.051034926 +0000 UTC m=+63.048051867" watchObservedRunningTime="2026-07-08 20:01:35.051966385 +0000 UTC m=+63.048983302" Jul 08 20:01:35.974623 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:35.974573 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:36.010770 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.010735 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9"] Jul 08 20:01:36.013483 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.013461 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.020567 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.020465 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 08 20:01:36.020748 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.020732 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 20:01:36.021804 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.021718 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 08 20:01:36.021804 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.021754 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 20:01:36.021931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.021727 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-s82kj\"" Jul 08 20:01:36.021931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.021857 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 20:01:36.025840 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.025819 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9"] Jul 08 20:01:36.071535 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.071492 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:36.095770 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.095730 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61819ca7-d0ea-4877-805b-d237f793ddd7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.095946 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.095823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptbnb\" (UniqueName: \"kubernetes.io/projected/61819ca7-d0ea-4877-805b-d237f793ddd7-kube-api-access-ptbnb\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.095946 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.095867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61819ca7-d0ea-4877-805b-d237f793ddd7-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.095946 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.095938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61819ca7-d0ea-4877-805b-d237f793ddd7-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.196447 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.196405 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptbnb\" (UniqueName: \"kubernetes.io/projected/61819ca7-d0ea-4877-805b-d237f793ddd7-kube-api-access-ptbnb\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.196624 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.196457 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61819ca7-d0ea-4877-805b-d237f793ddd7-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.196624 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.196520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61819ca7-d0ea-4877-805b-d237f793ddd7-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.196624 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.196553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61819ca7-d0ea-4877-805b-d237f793ddd7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.196795 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:36.196644 2573 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 08 20:01:36.196795 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:36.196714 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61819ca7-d0ea-4877-805b-d237f793ddd7-prometheus-operator-tls podName:61819ca7-d0ea-4877-805b-d237f793ddd7 nodeName:}" failed. No retries permitted until 2026-07-08 20:01:36.696693926 +0000 UTC m=+64.693710843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/61819ca7-d0ea-4877-805b-d237f793ddd7-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-w4fn9" (UID: "61819ca7-d0ea-4877-805b-d237f793ddd7") : secret "prometheus-operator-tls" not found Jul 08 20:01:36.197362 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.197336 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61819ca7-d0ea-4877-805b-d237f793ddd7-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.199199 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.199168 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61819ca7-d0ea-4877-805b-d237f793ddd7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.209550 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.209519 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptbnb\" (UniqueName: \"kubernetes.io/projected/61819ca7-d0ea-4877-805b-d237f793ddd7-kube-api-access-ptbnb\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.576052 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.576017 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:36.700796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.700757 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61819ca7-d0ea-4877-805b-d237f793ddd7-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.703517 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.703490 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/61819ca7-d0ea-4877-805b-d237f793ddd7-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-w4fn9\" (UID: \"61819ca7-d0ea-4877-805b-d237f793ddd7\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.924423 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.924390 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" Jul 08 20:01:36.942588 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:36.942559 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdwjg" Jul 08 20:01:37.120215 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:37.120179 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:37.220020 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:37.219951 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:37.519956 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:37.519926 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9"] Jul 08 20:01:37.523046 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:37.523023 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61819ca7_d0ea_4877_805b_d237f793ddd7.slice/crio-d341f6339a70a3218b6e5c9ba003f4e245eb71640603e7bc37cd289026dbdc18 WatchSource:0}: Error finding container d341f6339a70a3218b6e5c9ba003f4e245eb71640603e7bc37cd289026dbdc18: Status 404 returned error can't find the container with id d341f6339a70a3218b6e5c9ba003f4e245eb71640603e7bc37cd289026dbdc18 Jul 08 20:01:38.045359 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.045304 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" event={"ID":"61819ca7-d0ea-4877-805b-d237f793ddd7","Type":"ContainerStarted","Data":"d341f6339a70a3218b6e5c9ba003f4e245eb71640603e7bc37cd289026dbdc18"} Jul 08 20:01:38.046796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.046776 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9454698c-8cf8b" event={"ID":"a61507dc-6151-4502-8b12-5d1604a83316","Type":"ContainerStarted","Data":"631a387eda9f16c01d86c7f1b8f5baabf2b2019e1b6a69d5ebbdb3a183b82cdb"} Jul 08 20:01:38.048637 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.048616 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lmwhx" event={"ID":"d128df57-bf08-4aeb-9513-d64f27bcf180","Type":"ContainerStarted","Data":"16b16c0017bf3c13af422536638678aad19500c45adbb80e3cda5c78647a7b0d"} Jul 08 20:01:38.068145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.068110 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9454698c-8cf8b" podStartSLOduration=1.716478984 podStartE2EDuration="6.068096481s" totalImagesPullingTime="4.351617497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:38.067353105 +0000 UTC m=+66.064370040" watchObservedRunningTime="2026-07-08 20:01:38.068096481 +0000 UTC m=+66.065113419" Jul 08 20:01:38.092983 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.090499 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-lmwhx" podStartSLOduration=2.175661197 podStartE2EDuration="12.090482638s" totalImagesPullingTime="9.914821441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:01:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:38.088997555 +0000 UTC m=+66.086014492" watchObservedRunningTime="2026-07-08 20:01:38.090482638 +0000 UTC m=+66.087499576" Jul 08 20:01:38.173046 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.173015 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:38.274155 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.274118 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:38.315337 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.315223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:38.315488 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.315343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:38.318280 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.318252 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 20:01:38.318439 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.318284 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 20:01:38.328508 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.328479 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/938c8e6d-1bc6-432e-a7fd-f1c3da1bf189-original-pull-secret\") pod \"global-pull-secret-syncer-4jt2s\" (UID: \"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189\") " pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:38.328659 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.328570 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cb1ff8c-6dc6-4981-a487-b73961dbd578-metrics-certs\") pod \"network-metrics-daemon-5glh6\" (UID: \"2cb1ff8c-6dc6-4981-a487-b73961dbd578\") " pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:38.497510 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.497480 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zs499\"" Jul 08 20:01:38.501801 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.501782 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4jt2s" Jul 08 20:01:38.505826 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.505798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5glh6" Jul 08 20:01:38.516950 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.516916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:38.520582 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.520558 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 20:01:38.530319 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.530272 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 20:01:38.541233 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.541197 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4cz2\" (UniqueName: \"kubernetes.io/projected/ac575ae2-11c1-4daa-8d9b-4b675f53f832-kube-api-access-p4cz2\") pod \"network-check-target-8jwsw\" (UID: \"ac575ae2-11c1-4daa-8d9b-4b675f53f832\") " pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:38.641155 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.641114 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4jt2s"] Jul 08 20:01:38.644119 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:38.644080 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938c8e6d_1bc6_432e_a7fd_f1c3da1bf189.slice/crio-9c8b6868b755837a2608e0fa5fc8225d0cba3b179a1f19af7572402c1941fe3b WatchSource:0}: Error finding container 9c8b6868b755837a2608e0fa5fc8225d0cba3b179a1f19af7572402c1941fe3b: Status 404 returned error can't find the container with id 9c8b6868b755837a2608e0fa5fc8225d0cba3b179a1f19af7572402c1941fe3b Jul 08 20:01:38.659817 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.659789 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5glh6"] Jul 08 20:01:38.792783 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.792751 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqvk8\"" Jul 08 20:01:38.800604 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:38.800576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:38.957535 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:38.957504 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb1ff8c_6dc6_4981_a487_b73961dbd578.slice/crio-aac6748998e457c45dc02515895c1f153a1f1c32f68acc393e684b5e9fa7de63 WatchSource:0}: Error finding container aac6748998e457c45dc02515895c1f153a1f1c32f68acc393e684b5e9fa7de63: Status 404 returned error can't find the container with id aac6748998e457c45dc02515895c1f153a1f1c32f68acc393e684b5e9fa7de63 Jul 08 20:01:39.059400 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:39.059280 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4jt2s" event={"ID":"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189","Type":"ContainerStarted","Data":"9c8b6868b755837a2608e0fa5fc8225d0cba3b179a1f19af7572402c1941fe3b"} Jul 08 20:01:39.061211 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:39.061176 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5glh6" event={"ID":"2cb1ff8c-6dc6-4981-a487-b73961dbd578","Type":"ContainerStarted","Data":"aac6748998e457c45dc02515895c1f153a1f1c32f68acc393e684b5e9fa7de63"} Jul 08 20:01:39.063675 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:39.063597 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" event={"ID":"61819ca7-d0ea-4877-805b-d237f793ddd7","Type":"ContainerStarted","Data":"8584f2fa2cae654a8e18d0b66e0d82356689a6a53e3671f91bc03b5e7ca326fe"} Jul 08 20:01:39.095792 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:39.095762 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8jwsw"] Jul 08 20:01:39.099567 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:39.099539 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac575ae2_11c1_4daa_8d9b_4b675f53f832.slice/crio-6e77efa306e9a3a02fe5cf2b1de51d7657a1e224e6511a7255e2efd342eea7f8 WatchSource:0}: Error finding container 6e77efa306e9a3a02fe5cf2b1de51d7657a1e224e6511a7255e2efd342eea7f8: Status 404 returned error can't find the container with id 6e77efa306e9a3a02fe5cf2b1de51d7657a1e224e6511a7255e2efd342eea7f8 Jul 08 20:01:39.295287 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:39.295181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:40.069796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.069722 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jwsw" event={"ID":"ac575ae2-11c1-4daa-8d9b-4b675f53f832","Type":"ContainerStarted","Data":"6e77efa306e9a3a02fe5cf2b1de51d7657a1e224e6511a7255e2efd342eea7f8"} Jul 08 20:01:40.072711 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.072674 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" event={"ID":"61819ca7-d0ea-4877-805b-d237f793ddd7","Type":"ContainerStarted","Data":"cec526e9610670ee827d9c3717413f2ef8f3a7e1e4547eb0867efccc5d0acab7"} Jul 08 20:01:40.095186 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.094817 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-w4fn9" podStartSLOduration=3.63832998 podStartE2EDuration="5.09479162s" totalImagesPullingTime="1.45646164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:40.094408461 +0000 UTC m=+68.091425400" watchObservedRunningTime="2026-07-08 20:01:40.09479162 +0000 UTC m=+68.091808558" Jul 08 20:01:40.170326 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.170273 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-69c44ffc75-2h7dl"] Jul 08 20:01:40.172412 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.172387 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.181483 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.181459 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 08 20:01:40.188579 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.188536 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c44ffc75-2h7dl"] Jul 08 20:01:40.233744 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.233695 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-serving-cert\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.233935 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.233786 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvmhq\" (UniqueName: \"kubernetes.io/projected/9fe1c02c-0aff-4c17-bec0-034d3c442963-kube-api-access-fvmhq\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.233935 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.233843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-config\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.233935 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.233906 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-service-ca\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.234104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.233950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-oauth-config\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.234104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.233985 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-oauth-serving-cert\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.234104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.234052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-trusted-ca-bundle\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.335145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.335040 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-oauth-serving-cert\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.335145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.335094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-trusted-ca-bundle\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.335145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.335134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-serving-cert\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.335701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.335169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvmhq\" (UniqueName: \"kubernetes.io/projected/9fe1c02c-0aff-4c17-bec0-034d3c442963-kube-api-access-fvmhq\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.335701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.335215 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-config\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.335701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.335241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-service-ca\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.335701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.335281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-oauth-config\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.335972 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.335938 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-oauth-serving-cert\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.336427 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.336372 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-service-ca\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.336542 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.336524 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-config\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.337087 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.337062 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-trusted-ca-bundle\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.338364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.338295 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-oauth-config\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.338641 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.338598 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-serving-cert\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.347375 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.347322 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvmhq\" (UniqueName: \"kubernetes.io/projected/9fe1c02c-0aff-4c17-bec0-034d3c442963-kube-api-access-fvmhq\") pod \"console-69c44ffc75-2h7dl\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.484392 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.484358 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:40.649417 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:40.649319 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-69c44ffc75-2h7dl"] Jul 08 20:01:40.818790 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:40.818755 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe1c02c_0aff_4c17_bec0_034d3c442963.slice/crio-256edfd7ee290991e7d49f884af552614cdf8e3b747da95d35dd79de05195e5b WatchSource:0}: Error finding container 256edfd7ee290991e7d49f884af552614cdf8e3b747da95d35dd79de05195e5b: Status 404 returned error can't find the container with id 256edfd7ee290991e7d49f884af552614cdf8e3b747da95d35dd79de05195e5b Jul 08 20:01:41.078336 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.077917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5glh6" event={"ID":"2cb1ff8c-6dc6-4981-a487-b73961dbd578","Type":"ContainerStarted","Data":"4a81d0772c7ace80b84b297a88041c770247e205ae78045ad7cf35d8b86f1e58"} Jul 08 20:01:41.078336 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.077977 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5glh6" event={"ID":"2cb1ff8c-6dc6-4981-a487-b73961dbd578","Type":"ContainerStarted","Data":"77125df169950ca42be1cbce43a42ef537872ca0352b2029870fe63026685089"} Jul 08 20:01:41.079694 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.079668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c44ffc75-2h7dl" event={"ID":"9fe1c02c-0aff-4c17-bec0-034d3c442963","Type":"ContainerStarted","Data":"3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33"} Jul 08 20:01:41.079785 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.079712 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c44ffc75-2h7dl" event={"ID":"9fe1c02c-0aff-4c17-bec0-034d3c442963","Type":"ContainerStarted","Data":"256edfd7ee290991e7d49f884af552614cdf8e3b747da95d35dd79de05195e5b"} Jul 08 20:01:41.117928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.117801 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5glh6" podStartSLOduration=66.229534078 podStartE2EDuration="1m8.117768703s" totalImagesPullingTime="1.888234625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:41.097544022 +0000 UTC m=+69.094560960" watchObservedRunningTime="2026-07-08 20:01:41.117768703 +0000 UTC m=+69.114785653" Jul 08 20:01:41.118339 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.118272 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-69c44ffc75-2h7dl" podStartSLOduration=1.118260375 podStartE2EDuration="1.118260375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:41.117097372 +0000 UTC m=+69.114114372" watchObservedRunningTime="2026-07-08 20:01:41.118260375 +0000 UTC m=+69.115277313" Jul 08 20:01:41.214147 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.214111 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:41.281421 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.281377 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:41.580041 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.580005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:41.646161 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:41.646128 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:42.146343 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.146288 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:42.402723 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.402635 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:42.476378 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.476339 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn"] Jul 08 20:01:42.490956 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.490918 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.496479 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.496445 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 08 20:01:42.498885 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.497832 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 08 20:01:42.498885 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.497958 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5jfgx\"" Jul 08 20:01:42.499726 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.499704 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn"] Jul 08 20:01:42.515379 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.515346 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6lk7v"] Jul 08 20:01:42.529234 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.529204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:42.536822 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.536790 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx"] Jul 08 20:01:42.536977 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.536943 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.539940 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.539910 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hvx2f\"" Jul 08 20:01:42.540572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.540550 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 20:01:42.540852 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.540685 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 20:01:42.541617 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.541585 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 20:01:42.551091 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.551055 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx"] Jul 08 20:01:42.551222 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.551189 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.551580 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.551552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/629383ec-cbbe-4daf-bf3f-d13ed29b5440-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.551727 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.551640 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/629383ec-cbbe-4daf-bf3f-d13ed29b5440-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.551727 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.551680 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw2cr\" (UniqueName: \"kubernetes.io/projected/629383ec-cbbe-4daf-bf3f-d13ed29b5440-kube-api-access-cw2cr\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.551727 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.551712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/629383ec-cbbe-4daf-bf3f-d13ed29b5440-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.555163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.555135 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 08 20:01:42.555356 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.555132 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 08 20:01:42.555853 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.555830 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5nqcw\"" Jul 08 20:01:42.557881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.557860 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 08 20:01:42.652143 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652099 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/629383ec-cbbe-4daf-bf3f-d13ed29b5440-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652146 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-sys\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652181 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652209 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652272 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-wtmp\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652293 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652368 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/629383ec-cbbe-4daf-bf3f-d13ed29b5440-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-tls\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-textfile\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652437 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652480 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9r4l\" (UniqueName: \"kubernetes.io/projected/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-kube-api-access-r9r4l\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652535 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.652639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-metrics-client-ca\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.653347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4btcl\" (UniqueName: \"kubernetes.io/projected/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-api-access-4btcl\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.653347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/629383ec-cbbe-4daf-bf3f-d13ed29b5440-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.653347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-root\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.653347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.653347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.652875 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw2cr\" (UniqueName: \"kubernetes.io/projected/629383ec-cbbe-4daf-bf3f-d13ed29b5440-kube-api-access-cw2cr\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.653580 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.653471 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/629383ec-cbbe-4daf-bf3f-d13ed29b5440-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.654997 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.654974 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/629383ec-cbbe-4daf-bf3f-d13ed29b5440-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.655124 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.655105 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/629383ec-cbbe-4daf-bf3f-d13ed29b5440-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.675451 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.675416 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw2cr\" (UniqueName: \"kubernetes.io/projected/629383ec-cbbe-4daf-bf3f-d13ed29b5440-kube-api-access-cw2cr\") pod \"openshift-state-metrics-54cf96dfcf-qnfzn\" (UID: \"629383ec-cbbe-4daf-bf3f-d13ed29b5440\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.689080 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.689047 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:42.711992 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.711952 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:42.712188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.712037 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:01:42.713237 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.713202 2573 patch_prober.go:28] interesting pod/console-9454698c-8cf8b container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" start-of-body= Jul 08 20:01:42.713383 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.713280 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-9454698c-8cf8b" podUID="a61507dc-6151-4502-8b12-5d1604a83316" containerName="console" probeResult="failure" output="Get \"https://10.134.0.10:8443/health\": dial tcp 10.134.0.10:8443: connect: connection refused" Jul 08 20:01:42.753364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753322 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-root\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.753544 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.753544 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753396 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-root\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.753544 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753419 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-sys\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.753544 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753455 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-sys\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.753544 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753483 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.753544 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753510 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.753828 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753548 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-wtmp\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.753828 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753672 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-wtmp\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.753828 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753680 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.753828 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753713 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.753828 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-tls\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.753828 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753776 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-textfile\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.753828 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.754167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753846 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9r4l\" (UniqueName: \"kubernetes.io/projected/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-kube-api-access-r9r4l\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.754167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753874 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.754167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-metrics-client-ca\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.754167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.753925 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4btcl\" (UniqueName: \"kubernetes.io/projected/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-api-access-4btcl\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.754167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.754013 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.754167 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:42.754102 2573 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 08 20:01:42.754167 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:01:42.754153 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-tls podName:ebd1b6a4-141b-4e6c-9727-56fc32ed1660 nodeName:}" failed. No retries permitted until 2026-07-08 20:01:43.25413596 +0000 UTC m=+71.251152879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-prtvx" (UID: "ebd1b6a4-141b-4e6c-9727-56fc32ed1660") : secret "kube-state-metrics-tls" not found Jul 08 20:01:42.754576 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.754217 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.754576 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.754217 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.754576 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.754528 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-textfile\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.754936 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.754908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-metrics-client-ca\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.755033 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.754966 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-accelerators-collector-config\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.756883 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.756816 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.756883 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.756816 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.757058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.757014 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-node-exporter-tls\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.764826 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.764793 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9r4l\" (UniqueName: \"kubernetes.io/projected/2f6a1c34-f16e-4352-9877-ee72b6b71fa7-kube-api-access-r9r4l\") pod \"node-exporter-6lk7v\" (UID: \"2f6a1c34-f16e-4352-9877-ee72b6b71fa7\") " pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.775282 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.775255 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4btcl\" (UniqueName: \"kubernetes.io/projected/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-api-access-4btcl\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:42.802005 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.801954 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" Jul 08 20:01:42.805955 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.805931 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:42.853205 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.853173 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6lk7v" Jul 08 20:01:42.936160 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:42.936110 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:43.258066 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.257973 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:43.260614 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.260589 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/ebd1b6a4-141b-4e6c-9727-56fc32ed1660-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-prtvx\" (UID: \"ebd1b6a4-141b-4e6c-9727-56fc32ed1660\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:43.364467 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.364429 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:43.462796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.462760 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" Jul 08 20:01:43.476621 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.476580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:43.604066 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.603984 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:43.737522 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.737492 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 20:01:43.740272 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.740249 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.741142 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.741111 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:43.742848 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.742825 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 20:01:43.743016 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.742996 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 20:01:43.743512 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.743490 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 20:01:43.743635 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.743521 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 20:01:43.744526 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.743856 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 20:01:43.744526 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.743939 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 20:01:43.744526 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.744122 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 20:01:43.744526 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.744369 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 20:01:43.744771 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.744525 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 20:01:43.744822 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.744790 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4nzvq\"" Jul 08 20:01:43.766705 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.766671 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 20:01:43.863826 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.863740 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmrj\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-kube-api-access-jxmrj\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.863826 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.863802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864036 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.863901 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864036 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.863939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864036 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.863996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-web-config\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864036 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.864026 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.864050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-out\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.864073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.864100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.864129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-volume\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.864147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.864175 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.864209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.864201 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965270 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965482 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965290 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965482 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965330 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965482 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965378 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-web-config\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965482 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965482 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-out\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965494 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965519 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965547 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-volume\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965627 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.965739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965657 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmrj\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-kube-api-access-jxmrj\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.966110 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.965991 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.966956 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.966926 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.967075 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.966926 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.968889 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.968833 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.969360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.969338 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-out\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.969442 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.969414 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.970158 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.969832 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.970158 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.969957 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.970528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.970482 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.970528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.970500 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-volume\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.970688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.970638 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.971815 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.971797 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-web-config\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:43.974298 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:43.974273 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmrj\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-kube-api-access-jxmrj\") pod \"alertmanager-main-0\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:44.037425 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:44.037396 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v7nfw" Jul 08 20:01:44.054163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:44.054133 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:01:44.309821 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:44.309677 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6a1c34_f16e_4352_9877_ee72b6b71fa7.slice/crio-0b1be2272346ba81f1a8af1e31148690ded3f9b390d713d711ee2100f6b36e70 WatchSource:0}: Error finding container 0b1be2272346ba81f1a8af1e31148690ded3f9b390d713d711ee2100f6b36e70: Status 404 returned error can't find the container with id 0b1be2272346ba81f1a8af1e31148690ded3f9b390d713d711ee2100f6b36e70 Jul 08 20:01:44.490969 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:44.490876 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx"] Jul 08 20:01:44.494953 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:44.494913 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd1b6a4_141b_4e6c_9727_56fc32ed1660.slice/crio-8b57ce5046cacfd9e51a95a8655c3a068f1a87db242dc0ce821f8a0ae5218021 WatchSource:0}: Error finding container 8b57ce5046cacfd9e51a95a8655c3a068f1a87db242dc0ce821f8a0ae5218021: Status 404 returned error can't find the container with id 8b57ce5046cacfd9e51a95a8655c3a068f1a87db242dc0ce821f8a0ae5218021 Jul 08 20:01:44.512657 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:44.512607 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 20:01:44.515771 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:44.515742 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70df7ff4_8608_4f65_842e_00a8a78c7f8c.slice/crio-4ea91640b4a367dc9ff563f2644ab8a32c581334481ae12a074458442661dc4a WatchSource:0}: Error finding container 4ea91640b4a367dc9ff563f2644ab8a32c581334481ae12a074458442661dc4a: Status 404 returned error can't find the container with id 4ea91640b4a367dc9ff563f2644ab8a32c581334481ae12a074458442661dc4a Jul 08 20:01:44.529250 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:44.528952 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn"] Jul 08 20:01:44.532209 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:44.532169 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629383ec_cbbe_4daf_bf3f_d13ed29b5440.slice/crio-2396c7a8c92605cba80b5f8fa399209b3d96d85fcb7f7c30bcd4d5b893b47bcd WatchSource:0}: Error finding container 2396c7a8c92605cba80b5f8fa399209b3d96d85fcb7f7c30bcd4d5b893b47bcd: Status 404 returned error can't find the container with id 2396c7a8c92605cba80b5f8fa399209b3d96d85fcb7f7c30bcd4d5b893b47bcd Jul 08 20:01:45.098511 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.098469 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerStarted","Data":"4ea91640b4a367dc9ff563f2644ab8a32c581334481ae12a074458442661dc4a"} Jul 08 20:01:45.100580 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.100550 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" event={"ID":"629383ec-cbbe-4daf-bf3f-d13ed29b5440","Type":"ContainerStarted","Data":"4b878989c49febaf1e001fdd1bbc28569fb78f1b34781355229294a853058fcc"} Jul 08 20:01:45.100725 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.100611 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" event={"ID":"629383ec-cbbe-4daf-bf3f-d13ed29b5440","Type":"ContainerStarted","Data":"90109af8df5c7500d1d12b3397947af9a841a2031b68d986c6f5587ce9e5ed66"} Jul 08 20:01:45.100725 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.100642 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" event={"ID":"629383ec-cbbe-4daf-bf3f-d13ed29b5440","Type":"ContainerStarted","Data":"2396c7a8c92605cba80b5f8fa399209b3d96d85fcb7f7c30bcd4d5b893b47bcd"} Jul 08 20:01:45.101729 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.101702 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" event={"ID":"ebd1b6a4-141b-4e6c-9727-56fc32ed1660","Type":"ContainerStarted","Data":"8b57ce5046cacfd9e51a95a8655c3a068f1a87db242dc0ce821f8a0ae5218021"} Jul 08 20:01:45.103223 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.103201 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4jt2s" event={"ID":"938c8e6d-1bc6-432e-a7fd-f1c3da1bf189","Type":"ContainerStarted","Data":"143c5366b16134c80c1528af7f5861820293bc77b1b6d1a4a89ad0b0a93521bf"} Jul 08 20:01:45.104817 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.104794 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8jwsw" event={"ID":"ac575ae2-11c1-4daa-8d9b-4b675f53f832","Type":"ContainerStarted","Data":"b0dfeae1dd32469eaeda605c37bad73a174bd3e7faeb74504b113b3649764501"} Jul 08 20:01:45.104950 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.104934 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:01:45.106060 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.106035 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lk7v" event={"ID":"2f6a1c34-f16e-4352-9877-ee72b6b71fa7","Type":"ContainerStarted","Data":"0b1be2272346ba81f1a8af1e31148690ded3f9b390d713d711ee2100f6b36e70"} Jul 08 20:01:45.119628 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.119579 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4jt2s" podStartSLOduration=66.443363192 podStartE2EDuration="1m12.119563233s" totalImagesPullingTime="5.676200041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:45.118013347 +0000 UTC m=+73.115030282" watchObservedRunningTime="2026-07-08 20:01:45.119563233 +0000 UTC m=+73.116580208" Jul 08 20:01:45.136160 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.136102 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8jwsw" podStartSLOduration=66.936981092 podStartE2EDuration="1m12.136079579s" totalImagesPullingTime="5.199098487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:00:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:45.133910153 +0000 UTC m=+73.130927090" watchObservedRunningTime="2026-07-08 20:01:45.136079579 +0000 UTC m=+73.133096517" Jul 08 20:01:45.381185 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.381104 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:45.483101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.483067 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:45.490832 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.490803 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-68655c9874-zf74w"] Jul 08 20:01:45.493746 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.493724 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.497597 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.497569 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 08 20:01:45.497757 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.497740 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-6fla74a2b72ni\"" Jul 08 20:01:45.497757 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.497752 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-krf4c\"" Jul 08 20:01:45.497919 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.497889 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 08 20:01:45.498028 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.497951 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 08 20:01:45.498028 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.497963 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 08 20:01:45.498201 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.498157 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 08 20:01:45.513279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.513248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68655c9874-zf74w"] Jul 08 20:01:45.561515 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.561415 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:45.589022 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.584431 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe56b542-818b-468b-9d93-b8c12d6b0b9a-metrics-client-ca\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.589022 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.584485 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grvs5\" (UniqueName: \"kubernetes.io/projected/fe56b542-818b-468b-9d93-b8c12d6b0b9a-kube-api-access-grvs5\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.589022 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.584534 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.589022 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.584566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.589022 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.584622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.589022 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.584648 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-grpc-tls\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.589022 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.584702 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.589022 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.584736 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-tls\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.685487 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.685446 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.685665 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.685501 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-grpc-tls\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.685665 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.685553 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.685665 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.685590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-tls\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.685853 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.685797 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe56b542-818b-468b-9d93-b8c12d6b0b9a-metrics-client-ca\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.685974 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.685905 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grvs5\" (UniqueName: \"kubernetes.io/projected/fe56b542-818b-468b-9d93-b8c12d6b0b9a-kube-api-access-grvs5\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.685974 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.685956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.686449 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.686211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.686546 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.686517 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe56b542-818b-468b-9d93-b8c12d6b0b9a-metrics-client-ca\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.689170 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.689115 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.689275 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.689208 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.690132 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.690110 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.690616 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.690592 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-tls\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.691319 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.691283 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.692405 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.692383 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe56b542-818b-468b-9d93-b8c12d6b0b9a-secret-grpc-tls\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.696216 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.696174 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grvs5\" (UniqueName: \"kubernetes.io/projected/fe56b542-818b-468b-9d93-b8c12d6b0b9a-kube-api-access-grvs5\") pod \"thanos-querier-68655c9874-zf74w\" (UID: \"fe56b542-818b-468b-9d93-b8c12d6b0b9a\") " pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.805754 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.805716 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:45.993975 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:45.993938 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:46.058587 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:46.058534 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:46.112227 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:46.112186 2573 generic.go:356] "Generic (PLEG): container finished" podID="2f6a1c34-f16e-4352-9877-ee72b6b71fa7" containerID="6f8e14887edcaff71e5a642ee5c1d41be268539f62fe74ed231acce082b246e8" exitCode=0 Jul 08 20:01:46.112690 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:46.112254 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lk7v" event={"ID":"2f6a1c34-f16e-4352-9877-ee72b6b71fa7","Type":"ContainerDied","Data":"6f8e14887edcaff71e5a642ee5c1d41be268539f62fe74ed231acce082b246e8"} Jul 08 20:01:46.616825 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:46.616799 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-68655c9874-zf74w"] Jul 08 20:01:47.015020 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.014927 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-c6cfb6b5c-srf65"] Jul 08 20:01:47.017304 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.017281 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.021551 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.021527 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ae2oufiouc8h1\"" Jul 08 20:01:47.022774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.022665 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-87s98\"" Jul 08 20:01:47.022774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.022726 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 08 20:01:47.022774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.022741 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 08 20:01:47.022774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.022753 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 08 20:01:47.023030 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.022839 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 08 20:01:47.035528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.035357 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c6cfb6b5c-srf65"] Jul 08 20:01:47.103834 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.103798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ba3c834-b8c6-4574-8f48-cbbeb086589f-client-ca-bundle\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.103834 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.103840 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ba3c834-b8c6-4574-8f48-cbbeb086589f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.104064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.103862 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ba3c834-b8c6-4574-8f48-cbbeb086589f-metrics-server-audit-profiles\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.104064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.103899 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8mjs\" (UniqueName: \"kubernetes.io/projected/3ba3c834-b8c6-4574-8f48-cbbeb086589f-kube-api-access-z8mjs\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.104064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.103954 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ba3c834-b8c6-4574-8f48-cbbeb086589f-secret-metrics-server-client-certs\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.104064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.104005 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ba3c834-b8c6-4574-8f48-cbbeb086589f-secret-metrics-server-tls\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.104064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.104034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ba3c834-b8c6-4574-8f48-cbbeb086589f-audit-log\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.118804 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.118770 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lk7v" event={"ID":"2f6a1c34-f16e-4352-9877-ee72b6b71fa7","Type":"ContainerStarted","Data":"86f2d02cc7896c3e6b1ca03ddff319838a7c0a9b60bff776fe28f21c8e3c5633"} Jul 08 20:01:47.119251 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.118828 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6lk7v" event={"ID":"2f6a1c34-f16e-4352-9877-ee72b6b71fa7","Type":"ContainerStarted","Data":"aae5f3bd15b5b70c4c9d7602601a75b7b384b7041dcea62c754c23b16b91ed29"} Jul 08 20:01:47.120391 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.120367 2573 generic.go:356] "Generic (PLEG): container finished" podID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerID="a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492" exitCode=0 Jul 08 20:01:47.120485 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.120446 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerDied","Data":"a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492"} Jul 08 20:01:47.122606 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.122584 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" event={"ID":"629383ec-cbbe-4daf-bf3f-d13ed29b5440","Type":"ContainerStarted","Data":"01af2911787ebef536af7ce589b97128e84e74be5def80d61c6af930f4b81a01"} Jul 08 20:01:47.123852 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.123831 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" event={"ID":"fe56b542-818b-468b-9d93-b8c12d6b0b9a","Type":"ContainerStarted","Data":"401b6b6b55bf4b0140141cefca9f5180510f80e30153067bb17d746a3681b9dc"} Jul 08 20:01:47.125912 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.125896 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" event={"ID":"ebd1b6a4-141b-4e6c-9727-56fc32ed1660","Type":"ContainerStarted","Data":"c24ae29d38342e8fcd5b324fe7382e4756bfa0f90fe0577a55abfe802003b5ed"} Jul 08 20:01:47.126014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.125934 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" event={"ID":"ebd1b6a4-141b-4e6c-9727-56fc32ed1660","Type":"ContainerStarted","Data":"99f0e8d3ae6f960b4841b66c50943fe1b84a7e7e526bc633c042bbc34ab0b9c0"} Jul 08 20:01:47.126014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.125953 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" event={"ID":"ebd1b6a4-141b-4e6c-9727-56fc32ed1660","Type":"ContainerStarted","Data":"49164ce76432599f6a622a0bb912a36efcc2e5be34f70887a8fdb95cf4d97650"} Jul 08 20:01:47.166452 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.166390 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6lk7v" podStartSLOduration=3.969709576 podStartE2EDuration="5.166366292s" totalImagesPullingTime="1.106900939s" totalInitContainerRuntime="89.755777ms" isStatefulPod=true podCreationTimestamp="2026-07-08 20:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:47.165153023 +0000 UTC m=+75.162169961" watchObservedRunningTime="2026-07-08 20:01:47.166366292 +0000 UTC m=+75.163383233" Jul 08 20:01:47.204793 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.204760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ba3c834-b8c6-4574-8f48-cbbeb086589f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.204962 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.204821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ba3c834-b8c6-4574-8f48-cbbeb086589f-metrics-server-audit-profiles\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.204962 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.204869 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8mjs\" (UniqueName: \"kubernetes.io/projected/3ba3c834-b8c6-4574-8f48-cbbeb086589f-kube-api-access-z8mjs\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.205101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.205012 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ba3c834-b8c6-4574-8f48-cbbeb086589f-secret-metrics-server-client-certs\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.205639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.205613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ba3c834-b8c6-4574-8f48-cbbeb086589f-secret-metrics-server-tls\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.205831 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.205813 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ba3c834-b8c6-4574-8f48-cbbeb086589f-audit-log\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.205983 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.205739 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ba3c834-b8c6-4574-8f48-cbbeb086589f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.206058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.205983 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3ba3c834-b8c6-4574-8f48-cbbeb086589f-metrics-server-audit-profiles\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.206221 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.206197 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3ba3c834-b8c6-4574-8f48-cbbeb086589f-audit-log\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.206384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.206360 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ba3c834-b8c6-4574-8f48-cbbeb086589f-client-ca-bundle\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.207806 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.207782 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3ba3c834-b8c6-4574-8f48-cbbeb086589f-secret-metrics-server-client-certs\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.208469 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.208449 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3ba3c834-b8c6-4574-8f48-cbbeb086589f-client-ca-bundle\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.208798 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.208775 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3ba3c834-b8c6-4574-8f48-cbbeb086589f-secret-metrics-server-tls\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.219873 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.219845 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8mjs\" (UniqueName: \"kubernetes.io/projected/3ba3c834-b8c6-4574-8f48-cbbeb086589f-kube-api-access-z8mjs\") pod \"metrics-server-c6cfb6b5c-srf65\" (UID: \"3ba3c834-b8c6-4574-8f48-cbbeb086589f\") " pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.237232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.237204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:47.237594 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.237553 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-qnfzn" podStartSLOduration=3.435944383 podStartE2EDuration="5.237539711s" totalImagesPullingTime="1.801595328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:47.200781356 +0000 UTC m=+75.197798291" watchObservedRunningTime="2026-07-08 20:01:47.237539711 +0000 UTC m=+75.234556646" Jul 08 20:01:47.237667 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.237647 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-prtvx" podStartSLOduration=3.270128315 podStartE2EDuration="5.237642265s" totalImagesPullingTime="1.96751395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:47.234886958 +0000 UTC m=+75.231903893" watchObservedRunningTime="2026-07-08 20:01:47.237642265 +0000 UTC m=+75.234659200" Jul 08 20:01:47.249755 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.249727 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4"] Jul 08 20:01:47.252064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.252043 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:47.255750 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.255727 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4m9qf\"" Jul 08 20:01:47.255957 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.255931 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 08 20:01:47.280210 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.280142 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4"] Jul 08 20:01:47.326814 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.326782 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:01:47.408230 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.408192 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbg2l\" (UniqueName: \"kubernetes.io/projected/ed5f81ba-c6f4-42ed-aeb7-46e910fd9406-kube-api-access-xbg2l\") pod \"monitoring-plugin-d5dcff7cf-m67n4\" (UID: \"ed5f81ba-c6f4-42ed-aeb7-46e910fd9406\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:47.408415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.408303 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed5f81ba-c6f4-42ed-aeb7-46e910fd9406-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-m67n4\" (UID: \"ed5f81ba-c6f4-42ed-aeb7-46e910fd9406\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:47.430858 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.430823 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:47.472505 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.472473 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c6cfb6b5c-srf65"] Jul 08 20:01:47.475984 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:47.475951 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba3c834_b8c6_4574_8f48_cbbeb086589f.slice/crio-4a6016d5c5c6837df7bba81bacd82f4e48ed6e9d2694553afbd5cf82f892d672 WatchSource:0}: Error finding container 4a6016d5c5c6837df7bba81bacd82f4e48ed6e9d2694553afbd5cf82f892d672: Status 404 returned error can't find the container with id 4a6016d5c5c6837df7bba81bacd82f4e48ed6e9d2694553afbd5cf82f892d672 Jul 08 20:01:47.509101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.509062 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbg2l\" (UniqueName: \"kubernetes.io/projected/ed5f81ba-c6f4-42ed-aeb7-46e910fd9406-kube-api-access-xbg2l\") pod \"monitoring-plugin-d5dcff7cf-m67n4\" (UID: \"ed5f81ba-c6f4-42ed-aeb7-46e910fd9406\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:47.509292 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.509171 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed5f81ba-c6f4-42ed-aeb7-46e910fd9406-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-m67n4\" (UID: \"ed5f81ba-c6f4-42ed-aeb7-46e910fd9406\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:47.512054 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.512025 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ed5f81ba-c6f4-42ed-aeb7-46e910fd9406-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-m67n4\" (UID: \"ed5f81ba-c6f4-42ed-aeb7-46e910fd9406\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:47.518736 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.518703 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbg2l\" (UniqueName: \"kubernetes.io/projected/ed5f81ba-c6f4-42ed-aeb7-46e910fd9406-kube-api-access-xbg2l\") pod \"monitoring-plugin-d5dcff7cf-m67n4\" (UID: \"ed5f81ba-c6f4-42ed-aeb7-46e910fd9406\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:47.526204 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.526172 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:47.562829 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.562735 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:47.714128 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.714051 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4"] Jul 08 20:01:47.717988 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:47.717953 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5f81ba_c6f4_42ed_aeb7_46e910fd9406.slice/crio-9441a5d1119c95137ba123a44a403b0968bb545294cf3cabe71f1346833a4b2a WatchSource:0}: Error finding container 9441a5d1119c95137ba123a44a403b0968bb545294cf3cabe71f1346833a4b2a: Status 404 returned error can't find the container with id 9441a5d1119c95137ba123a44a403b0968bb545294cf3cabe71f1346833a4b2a Jul 08 20:01:47.801798 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.801767 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9454698c-8cf8b"] Jul 08 20:01:47.831280 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.831214 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cd786c966-2s9w8"] Jul 08 20:01:47.833790 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.833763 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:47.848266 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.848245 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd786c966-2s9w8"] Jul 08 20:01:47.912803 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.912755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-serving-cert\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:47.912978 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.912826 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-oauth-serving-cert\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:47.912978 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.912908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l8g7\" (UniqueName: \"kubernetes.io/projected/250580a7-f77b-4f07-a286-231398b0a786-kube-api-access-2l8g7\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:47.912978 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.912939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-oauth-config\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:47.912978 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.912975 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-service-ca\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:47.913182 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.913051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-console-config\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:47.913182 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:47.913117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-trusted-ca-bundle\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.013788 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.013758 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-serving-cert\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.013944 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.013808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-oauth-serving-cert\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.013944 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.013926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l8g7\" (UniqueName: \"kubernetes.io/projected/250580a7-f77b-4f07-a286-231398b0a786-kube-api-access-2l8g7\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.014064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.013961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-oauth-config\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.014064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.013986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-service-ca\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.014064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.014008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-console-config\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.014203 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.014110 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-trusted-ca-bundle\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.014578 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.014553 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-oauth-serving-cert\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.014694 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.014679 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-console-config\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.014757 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.014682 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-service-ca\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.014954 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.014935 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-trusted-ca-bundle\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.016162 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.016142 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-oauth-config\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.016269 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.016246 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-serving-cert\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.035156 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.035131 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l8g7\" (UniqueName: \"kubernetes.io/projected/250580a7-f77b-4f07-a286-231398b0a786-kube-api-access-2l8g7\") pod \"console-7cd786c966-2s9w8\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.132743 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.132660 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" event={"ID":"ed5f81ba-c6f4-42ed-aeb7-46e910fd9406","Type":"ContainerStarted","Data":"9441a5d1119c95137ba123a44a403b0968bb545294cf3cabe71f1346833a4b2a"} Jul 08 20:01:48.134129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.134094 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" event={"ID":"3ba3c834-b8c6-4574-8f48-cbbeb086589f","Type":"ContainerStarted","Data":"4a6016d5c5c6837df7bba81bacd82f4e48ed6e9d2694553afbd5cf82f892d672"} Jul 08 20:01:48.146172 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.146152 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:48.315917 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.315846 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cd786c966-2s9w8"] Jul 08 20:01:48.817512 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.817474 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:48.824780 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.824742 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 20:01:48.828832 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.828592 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.836028 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.836001 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 20:01:48.836162 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.836043 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 20:01:48.836162 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.836089 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 20:01:48.836162 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.836043 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 20:01:48.837494 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.836445 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 20:01:48.841420 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.838419 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 20:01:48.841420 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.838961 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 20:01:48.842434 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.842412 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 20:01:48.843047 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.842838 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 20:01:48.843104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.843088 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9su9qmk9e6k2v\"" Jul 08 20:01:48.843354 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.843338 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 20:01:48.845995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.845969 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 20:01:48.846801 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.846782 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-prldp\"" Jul 08 20:01:48.847254 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.847238 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 20:01:48.852536 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.850598 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 20:01:48.856049 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.856022 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 20:01:48.866822 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.866794 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 20:01:48.923520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923252 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923379 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923409 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923458 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923481 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config-out\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923776 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923798 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jzd2\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-kube-api-access-4jzd2\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923820 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923916 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923957 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.923999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:48.923974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-web-config\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:48.973369 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:48.973339 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250580a7_f77b_4f07_a286_231398b0a786.slice/crio-f0b6a0e8632791489633671b3fb3fed45011a4a53fd3dec7074f6c1331023b66 WatchSource:0}: Error finding container f0b6a0e8632791489633671b3fb3fed45011a4a53fd3dec7074f6c1331023b66: Status 404 returned error can't find the container with id f0b6a0e8632791489633671b3fb3fed45011a4a53fd3dec7074f6c1331023b66 Jul 08 20:01:49.025463 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.025696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025473 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.025696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025505 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.025696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-web-config\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.025696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025576 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.025696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.025696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025625 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.025696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025653 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.025696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025677 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.025696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025699 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025722 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025800 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config-out\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025889 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025911 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025962 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jzd2\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-kube-api-access-4jzd2\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.025990 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.026014 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026642 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.026356 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.026642 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.026404 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.029712 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.028911 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.029712 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.028981 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.029712 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.029279 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.029712 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.029430 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.030071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.029727 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.031096 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.030775 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.031714 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.031691 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.032654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.032613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.032979 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.032954 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-web-config\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.034598 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.034574 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config-out\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.035175 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.035134 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.036438 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.036415 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.036525 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.036500 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.037225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.035838 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.037225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.037189 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.038058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.038019 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.039145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.039036 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.045468 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.045448 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jzd2\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-kube-api-access-4jzd2\") pod \"prometheus-k8s-0\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:49.140412 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.140325 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd786c966-2s9w8" event={"ID":"250580a7-f77b-4f07-a286-231398b0a786","Type":"ContainerStarted","Data":"f0b6a0e8632791489633671b3fb3fed45011a4a53fd3dec7074f6c1331023b66"} Jul 08 20:01:49.153708 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:49.153682 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:50.249467 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:50.249440 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 20:01:50.252400 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:01:50.252371 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f2b5db_c8a4_4c25_87db_f94cb6f6bd99.slice/crio-0389cbc2753cece171e581c70ad47e8ca48102a7c95a45a105f5a242e94852b9 WatchSource:0}: Error finding container 0389cbc2753cece171e581c70ad47e8ca48102a7c95a45a105f5a242e94852b9: Status 404 returned error can't find the container with id 0389cbc2753cece171e581c70ad47e8ca48102a7c95a45a105f5a242e94852b9 Jul 08 20:01:50.484616 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:50.484589 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:50.484743 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:50.484677 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:50.489493 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:50.489470 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:51.152990 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.152953 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" event={"ID":"ed5f81ba-c6f4-42ed-aeb7-46e910fd9406","Type":"ContainerStarted","Data":"99145b691d1d157a33af9e28401b09f52396215328b38971bc02c4df0ad99b93"} Jul 08 20:01:51.153184 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.153167 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:51.156585 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.156561 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerStarted","Data":"092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d"} Jul 08 20:01:51.156700 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.156626 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerStarted","Data":"6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1"} Jul 08 20:01:51.156700 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.156666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerStarted","Data":"cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8"} Jul 08 20:01:51.156784 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.156703 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerStarted","Data":"ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440"} Jul 08 20:01:51.156784 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.156740 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerStarted","Data":"4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871"} Jul 08 20:01:51.158593 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.158573 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" Jul 08 20:01:51.158810 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.158787 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd786c966-2s9w8" event={"ID":"250580a7-f77b-4f07-a286-231398b0a786","Type":"ContainerStarted","Data":"ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55"} Jul 08 20:01:51.160425 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.160402 2573 generic.go:356] "Generic (PLEG): container finished" podID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerID="18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49" exitCode=0 Jul 08 20:01:51.160529 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.160490 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerDied","Data":"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49"} Jul 08 20:01:51.160576 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.160547 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerStarted","Data":"0389cbc2753cece171e581c70ad47e8ca48102a7c95a45a105f5a242e94852b9"} Jul 08 20:01:51.163726 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.163702 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" event={"ID":"fe56b542-818b-468b-9d93-b8c12d6b0b9a","Type":"ContainerStarted","Data":"b2be6b19da84ac539e88b345b89c85144c12b8230f083508bdae27f05763b715"} Jul 08 20:01:51.163830 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.163770 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" event={"ID":"fe56b542-818b-468b-9d93-b8c12d6b0b9a","Type":"ContainerStarted","Data":"22c7883ce65567197fec5e9f98eefe8adcdf5392c95e132001e0d8333f6224e3"} Jul 08 20:01:51.163830 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.163813 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" event={"ID":"fe56b542-818b-468b-9d93-b8c12d6b0b9a","Type":"ContainerStarted","Data":"7fccb04ba9fb3900a0d410aacde226d641d1b4e1aa25effacaa0753695a9221c"} Jul 08 20:01:51.165834 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.165812 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" event={"ID":"3ba3c834-b8c6-4574-8f48-cbbeb086589f","Type":"ContainerStarted","Data":"35f5c76ffb43644f96d91ee9a21f2926285b05c7fe0071988d51e9efe3ea7da6"} Jul 08 20:01:51.171274 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.171255 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:01:51.176138 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.176088 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-m67n4" podStartSLOduration=1.808528877 podStartE2EDuration="4.176070975s" totalImagesPullingTime="2.367542098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:51.17386128 +0000 UTC m=+79.170878217" watchObservedRunningTime="2026-07-08 20:01:51.176070975 +0000 UTC m=+79.173087914" Jul 08 20:01:51.253727 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.253677 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" podStartSLOduration=2.645395563 podStartE2EDuration="5.253656502s" totalImagesPullingTime="2.608260939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:51.252727267 +0000 UTC m=+79.249744203" watchObservedRunningTime="2026-07-08 20:01:51.253656502 +0000 UTC m=+79.250673441" Jul 08 20:01:51.318934 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.318876 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cd786c966-2s9w8" podStartSLOduration=4.318855721 podStartE2EDuration="4.318855721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:51.317944306 +0000 UTC m=+79.314961244" watchObservedRunningTime="2026-07-08 20:01:51.318855721 +0000 UTC m=+79.315872637" Jul 08 20:01:51.383700 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.383642 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:51.482552 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:51.482526 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:52.175141 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:52.175105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerStarted","Data":"6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf"} Jul 08 20:01:52.178617 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:52.178559 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" event={"ID":"fe56b542-818b-468b-9d93-b8c12d6b0b9a","Type":"ContainerStarted","Data":"e1d2a452b81e6b1cb18f137222382251e75b38a3640cf66f9fed30f08c78b892"} Jul 08 20:01:52.178767 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:52.178636 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" event={"ID":"fe56b542-818b-468b-9d93-b8c12d6b0b9a","Type":"ContainerStarted","Data":"7f36456c66c488a9cb3a99f1efcdb56c578218b934986d70550cd1f4616fc2c6"} Jul 08 20:01:52.178767 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:52.178678 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" event={"ID":"fe56b542-818b-468b-9d93-b8c12d6b0b9a","Type":"ContainerStarted","Data":"023c1f8a2ef5ecd0fc016b46d7c8c706e147d87f9dec558b888f9e10a2e4b68e"} Jul 08 20:01:52.205962 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:52.205912 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.413958198 podStartE2EDuration="9.2058967s" totalImagesPullingTime="5.757926419s" totalInitContainerRuntime="34.012083ms" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:52.203125568 +0000 UTC m=+80.200142504" watchObservedRunningTime="2026-07-08 20:01:52.2058967 +0000 UTC m=+80.202913635" Jul 08 20:01:52.234063 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:52.234008 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" podStartSLOduration=2.8737128849999998 podStartE2EDuration="7.233987636s" totalImagesPullingTime="4.360274751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:52.233607963 +0000 UTC m=+80.230624900" watchObservedRunningTime="2026-07-08 20:01:52.233987636 +0000 UTC m=+80.231004573" Jul 08 20:01:53.182968 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:53.182933 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:01:53.307960 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:53.307924 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:53.655179 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:53.655088 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:53.752204 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:53.752167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:54.454644 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:54.454511 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:54.524820 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:54.524785 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:56.199489 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:56.199455 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerStarted","Data":"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e"} Jul 08 20:01:56.199931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:56.199544 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerStarted","Data":"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6"} Jul 08 20:01:56.199931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:56.199573 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerStarted","Data":"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079"} Jul 08 20:01:56.199931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:56.199599 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerStarted","Data":"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734"} Jul 08 20:01:56.199931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:56.199623 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerStarted","Data":"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba"} Jul 08 20:01:56.199931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:56.199644 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerStarted","Data":"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590"} Jul 08 20:01:56.239177 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:56.239114 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.104712931 podStartE2EDuration="8.23909265s" totalImagesPullingTime="4.1053353s" totalInitContainerRuntime="29.044419ms" isStatefulPod=false podCreationTimestamp="2026-07-08 20:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:01:56.236292063 +0000 UTC m=+84.233309004" watchObservedRunningTime="2026-07-08 20:01:56.23909265 +0000 UTC m=+84.236109588" Jul 08 20:01:58.147291 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:58.147256 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:58.147673 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:58.147350 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:58.152014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:58.151993 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:58.161212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:58.161191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:58.211823 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:58.211794 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:01:58.232885 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:58.232859 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:58.263734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:58.263708 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c44ffc75-2h7dl"] Jul 08 20:01:58.409052 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:58.408965 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:58.484444 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:58.484418 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:01:59.153902 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:59.153872 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:01:59.192833 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:01:59.192806 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-68655c9874-zf74w" Jul 08 20:02:00.294676 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:00.294623 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:00.396001 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:00.395969 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:01.548756 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:01.548725 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:01.649375 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:01.649327 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:01.750361 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:01.750332 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:02.261224 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:02.261191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:02.327800 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:02.327770 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:06.942723 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:06.942695 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:07.048282 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:07.048255 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:07.327294 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:07.327214 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:02:07.327294 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:07.327267 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:02:09.370927 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:09.370894 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:09.470712 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:09.470687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:09.578605 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:09.578577 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:11.257491 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:11.257465 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:11.317536 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:11.317510 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:12.829725 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:12.829682 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-9454698c-8cf8b" podUID="a61507dc-6151-4502-8b12-5d1604a83316" containerName="console" containerID="cri-o://631a387eda9f16c01d86c7f1b8f5baabf2b2019e1b6a69d5ebbdb3a183b82cdb" gracePeriod=15 Jul 08 20:02:13.277850 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.277816 2573 generic.go:356] "Generic (PLEG): container finished" podID="a61507dc-6151-4502-8b12-5d1604a83316" containerID="631a387eda9f16c01d86c7f1b8f5baabf2b2019e1b6a69d5ebbdb3a183b82cdb" exitCode=0 Jul 08 20:02:13.278027 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.277872 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9454698c-8cf8b" event={"ID":"a61507dc-6151-4502-8b12-5d1604a83316","Type":"ContainerDied","Data":"631a387eda9f16c01d86c7f1b8f5baabf2b2019e1b6a69d5ebbdb3a183b82cdb"} Jul 08 20:02:13.603147 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.603116 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:02:13.650319 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.650275 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-oauth-serving-cert\") pod \"a61507dc-6151-4502-8b12-5d1604a83316\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " Jul 08 20:02:13.650496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.650375 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-console-config\") pod \"a61507dc-6151-4502-8b12-5d1604a83316\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " Jul 08 20:02:13.650496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.650402 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-service-ca\") pod \"a61507dc-6151-4502-8b12-5d1604a83316\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " Jul 08 20:02:13.650496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.650457 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-serving-cert\") pod \"a61507dc-6151-4502-8b12-5d1604a83316\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " Jul 08 20:02:13.650496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.650478 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-oauth-config\") pod \"a61507dc-6151-4502-8b12-5d1604a83316\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " Jul 08 20:02:13.650711 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.650507 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4msx4\" (UniqueName: \"kubernetes.io/projected/a61507dc-6151-4502-8b12-5d1604a83316-kube-api-access-4msx4\") pod \"a61507dc-6151-4502-8b12-5d1604a83316\" (UID: \"a61507dc-6151-4502-8b12-5d1604a83316\") " Jul 08 20:02:13.650818 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.650785 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a61507dc-6151-4502-8b12-5d1604a83316" (UID: "a61507dc-6151-4502-8b12-5d1604a83316"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:02:13.650946 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.650790 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-service-ca" (OuterVolumeSpecName: "service-ca") pod "a61507dc-6151-4502-8b12-5d1604a83316" (UID: "a61507dc-6151-4502-8b12-5d1604a83316"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:02:13.650946 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.650825 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-console-config" (OuterVolumeSpecName: "console-config") pod "a61507dc-6151-4502-8b12-5d1604a83316" (UID: "a61507dc-6151-4502-8b12-5d1604a83316"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:02:13.652842 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.652796 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a61507dc-6151-4502-8b12-5d1604a83316" (UID: "a61507dc-6151-4502-8b12-5d1604a83316"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:02:13.652947 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.652845 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a61507dc-6151-4502-8b12-5d1604a83316" (UID: "a61507dc-6151-4502-8b12-5d1604a83316"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:02:13.652947 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.652845 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61507dc-6151-4502-8b12-5d1604a83316-kube-api-access-4msx4" (OuterVolumeSpecName: "kube-api-access-4msx4") pod "a61507dc-6151-4502-8b12-5d1604a83316" (UID: "a61507dc-6151-4502-8b12-5d1604a83316"). InnerVolumeSpecName "kube-api-access-4msx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:02:13.751765 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.751725 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:13.751765 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.751767 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a61507dc-6151-4502-8b12-5d1604a83316-console-oauth-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:13.751960 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.751782 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4msx4\" (UniqueName: \"kubernetes.io/projected/a61507dc-6151-4502-8b12-5d1604a83316-kube-api-access-4msx4\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:13.751960 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.751798 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-oauth-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:13.751960 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.751812 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-console-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:13.751960 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.751824 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a61507dc-6151-4502-8b12-5d1604a83316-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:13.841165 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.841073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:13.939282 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:13.939249 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:14.284615 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:14.284592 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9454698c-8cf8b" Jul 08 20:02:14.284800 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:14.284589 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9454698c-8cf8b" event={"ID":"a61507dc-6151-4502-8b12-5d1604a83316","Type":"ContainerDied","Data":"9a485228ae7d553554f7edf8ba81101a0eb13260709dc3b52d968b1df8e15ef7"} Jul 08 20:02:14.284800 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:14.284736 2573 scope.go:117] "RemoveContainer" containerID="631a387eda9f16c01d86c7f1b8f5baabf2b2019e1b6a69d5ebbdb3a183b82cdb" Jul 08 20:02:14.311850 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:14.311820 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9454698c-8cf8b"] Jul 08 20:02:14.317771 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:14.317741 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9454698c-8cf8b"] Jul 08 20:02:14.785706 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:14.785675 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a61507dc-6151-4502-8b12-5d1604a83316" path="/var/lib/kubelet/pods/a61507dc-6151-4502-8b12-5d1604a83316/volumes" Jul 08 20:02:16.114904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:16.114877 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8jwsw" Jul 08 20:02:23.284042 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.283994 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-69c44ffc75-2h7dl" podUID="9fe1c02c-0aff-4c17-bec0-034d3c442963" containerName="console" containerID="cri-o://3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33" gracePeriod=15 Jul 08 20:02:23.536887 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.536829 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:02:23.639752 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.639723 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-service-ca\") pod \"9fe1c02c-0aff-4c17-bec0-034d3c442963\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " Jul 08 20:02:23.639942 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.639806 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-serving-cert\") pod \"9fe1c02c-0aff-4c17-bec0-034d3c442963\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " Jul 08 20:02:23.639942 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.639830 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-oauth-config\") pod \"9fe1c02c-0aff-4c17-bec0-034d3c442963\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " Jul 08 20:02:23.639942 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.639869 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-oauth-serving-cert\") pod \"9fe1c02c-0aff-4c17-bec0-034d3c442963\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " Jul 08 20:02:23.639942 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.639907 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-trusted-ca-bundle\") pod \"9fe1c02c-0aff-4c17-bec0-034d3c442963\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " Jul 08 20:02:23.640158 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.639955 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvmhq\" (UniqueName: \"kubernetes.io/projected/9fe1c02c-0aff-4c17-bec0-034d3c442963-kube-api-access-fvmhq\") pod \"9fe1c02c-0aff-4c17-bec0-034d3c442963\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " Jul 08 20:02:23.640158 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.639978 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-config\") pod \"9fe1c02c-0aff-4c17-bec0-034d3c442963\" (UID: \"9fe1c02c-0aff-4c17-bec0-034d3c442963\") " Jul 08 20:02:23.640158 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.640064 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-service-ca" (OuterVolumeSpecName: "service-ca") pod "9fe1c02c-0aff-4c17-bec0-034d3c442963" (UID: "9fe1c02c-0aff-4c17-bec0-034d3c442963"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:02:23.640342 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.640216 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:23.640405 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.640346 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9fe1c02c-0aff-4c17-bec0-034d3c442963" (UID: "9fe1c02c-0aff-4c17-bec0-034d3c442963"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:02:23.640405 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.640358 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9fe1c02c-0aff-4c17-bec0-034d3c442963" (UID: "9fe1c02c-0aff-4c17-bec0-034d3c442963"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:02:23.640488 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.640417 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-config" (OuterVolumeSpecName: "console-config") pod "9fe1c02c-0aff-4c17-bec0-034d3c442963" (UID: "9fe1c02c-0aff-4c17-bec0-034d3c442963"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:02:23.642097 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.642067 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9fe1c02c-0aff-4c17-bec0-034d3c442963" (UID: "9fe1c02c-0aff-4c17-bec0-034d3c442963"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:02:23.642195 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.642074 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe1c02c-0aff-4c17-bec0-034d3c442963-kube-api-access-fvmhq" (OuterVolumeSpecName: "kube-api-access-fvmhq") pod "9fe1c02c-0aff-4c17-bec0-034d3c442963" (UID: "9fe1c02c-0aff-4c17-bec0-034d3c442963"). InnerVolumeSpecName "kube-api-access-fvmhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:02:23.642195 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.642143 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9fe1c02c-0aff-4c17-bec0-034d3c442963" (UID: "9fe1c02c-0aff-4c17-bec0-034d3c442963"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:02:23.740662 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.740632 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvmhq\" (UniqueName: \"kubernetes.io/projected/9fe1c02c-0aff-4c17-bec0-034d3c442963-kube-api-access-fvmhq\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:23.740662 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.740660 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:23.740662 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.740669 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:23.740865 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.740678 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fe1c02c-0aff-4c17-bec0-034d3c442963-console-oauth-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:23.740865 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.740687 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-oauth-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:23.740865 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:23.740695 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fe1c02c-0aff-4c17-bec0-034d3c442963-trusted-ca-bundle\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:02:24.342494 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.342454 2573 generic.go:356] "Generic (PLEG): container finished" podID="9fe1c02c-0aff-4c17-bec0-034d3c442963" containerID="3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33" exitCode=0 Jul 08 20:02:24.342494 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.342493 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-69c44ffc75-2h7dl" Jul 08 20:02:24.342949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.342539 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c44ffc75-2h7dl" event={"ID":"9fe1c02c-0aff-4c17-bec0-034d3c442963","Type":"ContainerDied","Data":"3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33"} Jul 08 20:02:24.342949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.342592 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-69c44ffc75-2h7dl" event={"ID":"9fe1c02c-0aff-4c17-bec0-034d3c442963","Type":"ContainerDied","Data":"256edfd7ee290991e7d49f884af552614cdf8e3b747da95d35dd79de05195e5b"} Jul 08 20:02:24.342949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.342613 2573 scope.go:117] "RemoveContainer" containerID="3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33" Jul 08 20:02:24.351078 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.351059 2573 scope.go:117] "RemoveContainer" containerID="3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33" Jul 08 20:02:24.351342 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:02:24.351322 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33\": container with ID starting with 3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33 not found: ID does not exist" containerID="3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33" Jul 08 20:02:24.351402 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.351351 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33"} err="failed to get container status \"3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33\": rpc error: code = NotFound desc = could not find container \"3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33\": container with ID starting with 3ba39f478a84ee4d969810b2c5e7c7f9d3c7e1b0acb27258626a3194b3d02c33 not found: ID does not exist" Jul 08 20:02:24.366912 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.366888 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-69c44ffc75-2h7dl"] Jul 08 20:02:24.373103 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.373084 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-69c44ffc75-2h7dl"] Jul 08 20:02:24.460546 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.460516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:24.559332 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.559277 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:24.785120 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:24.785089 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fe1c02c-0aff-4c17-bec0-034d3c442963" path="/var/lib/kubelet/pods/9fe1c02c-0aff-4c17-bec0-034d3c442963/volumes" Jul 08 20:02:27.332891 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:27.332861 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:02:27.336840 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:27.336821 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-c6cfb6b5c-srf65" Jul 08 20:02:49.154377 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:49.154344 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:02:49.175072 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:49.175049 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:02:49.460558 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:49.460536 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:02:50.810046 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:50.810001 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:02:50.888894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:02:50.888866 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:01.712676 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:01.712640 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:01.856579 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:01.856548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:02.797902 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:02.797868 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 20:03:02.798364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:02.798143 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="alertmanager" containerID="cri-o://4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871" gracePeriod=120 Jul 08 20:03:02.798364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:02.798210 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy-metric" containerID="cri-o://092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d" gracePeriod=120 Jul 08 20:03:02.798364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:02.798229 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy" containerID="cri-o://6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1" gracePeriod=120 Jul 08 20:03:02.798364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:02.798283 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="prom-label-proxy" containerID="cri-o://6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf" gracePeriod=120 Jul 08 20:03:02.798364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:02.798214 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy-web" containerID="cri-o://cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8" gracePeriod=120 Jul 08 20:03:02.798364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:02.798284 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="config-reloader" containerID="cri-o://ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440" gracePeriod=120 Jul 08 20:03:03.505689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:03.505655 2573 generic.go:356] "Generic (PLEG): container finished" podID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerID="6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf" exitCode=0 Jul 08 20:03:03.505689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:03.505682 2573 generic.go:356] "Generic (PLEG): container finished" podID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerID="6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1" exitCode=0 Jul 08 20:03:03.505689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:03.505690 2573 generic.go:356] "Generic (PLEG): container finished" podID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerID="ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440" exitCode=0 Jul 08 20:03:03.505689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:03.505696 2573 generic.go:356] "Generic (PLEG): container finished" podID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerID="4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871" exitCode=0 Jul 08 20:03:03.505967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:03.505726 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerDied","Data":"6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf"} Jul 08 20:03:03.505967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:03.505825 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerDied","Data":"6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1"} Jul 08 20:03:03.505967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:03.505854 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerDied","Data":"ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440"} Jul 08 20:03:03.505967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:03.505880 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerDied","Data":"4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871"} Jul 08 20:03:04.037129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.037107 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.068319 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068294 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-web-config\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.068437 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068358 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-trusted-ca-bundle\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.068437 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068390 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-tls-assets\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.068437 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068425 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.068599 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068452 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-volume\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.068599 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068510 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-metrics-client-ca\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.068599 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068542 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxmrj\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-kube-api-access-jxmrj\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.068599 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068571 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-main-tls\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.069331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068618 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.069331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068644 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-web\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.069331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068679 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-main-db\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.069331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068718 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-cluster-tls-config\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.069331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068753 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-out\") pod \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\" (UID: \"70df7ff4-8608-4f65-842e-00a8a78c7f8c\") " Jul 08 20:03:04.069331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.068761 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:04.069331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.069009 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.069331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.069150 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:03:04.069331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.069176 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:04.073133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.072946 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-out" (OuterVolumeSpecName: "config-out") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:03:04.073819 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.073777 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-kube-api-access-jxmrj" (OuterVolumeSpecName: "kube-api-access-jxmrj") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "kube-api-access-jxmrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:03:04.074721 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.074652 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:03:04.074721 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.074657 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:04.074879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.074859 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:04.074979 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.074954 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:04.075037 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.075011 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-volume" (OuterVolumeSpecName: "config-volume") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:04.075168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.075148 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:04.079582 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.079537 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:04.086992 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.086967 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-web-config" (OuterVolumeSpecName: "web-config") pod "70df7ff4-8608-4f65-842e-00a8a78c7f8c" (UID: "70df7ff4-8608-4f65-842e-00a8a78c7f8c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:04.170068 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170046 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-web-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170068 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170068 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-tls-assets\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170078 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170087 2573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-volume\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170095 2573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70df7ff4-8608-4f65-842e-00a8a78c7f8c-metrics-client-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170105 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxmrj\" (UniqueName: \"kubernetes.io/projected/70df7ff4-8608-4f65-842e-00a8a78c7f8c-kube-api-access-jxmrj\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170114 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-main-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170123 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170132 2573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170141 2573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-alertmanager-main-db\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170149 2573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70df7ff4-8608-4f65-842e-00a8a78c7f8c-cluster-tls-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.170217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.170157 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70df7ff4-8608-4f65-842e-00a8a78c7f8c-config-out\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:04.512774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.512698 2573 generic.go:356] "Generic (PLEG): container finished" podID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerID="092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d" exitCode=0 Jul 08 20:03:04.512774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.512719 2573 generic.go:356] "Generic (PLEG): container finished" podID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerID="cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8" exitCode=0 Jul 08 20:03:04.512774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.512752 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerDied","Data":"092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d"} Jul 08 20:03:04.513028 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.512784 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.513028 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.512807 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerDied","Data":"cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8"} Jul 08 20:03:04.513028 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.512846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70df7ff4-8608-4f65-842e-00a8a78c7f8c","Type":"ContainerDied","Data":"4ea91640b4a367dc9ff563f2644ab8a32c581334481ae12a074458442661dc4a"} Jul 08 20:03:04.513028 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.512879 2573 scope.go:117] "RemoveContainer" containerID="6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf" Jul 08 20:03:04.520209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.520187 2573 scope.go:117] "RemoveContainer" containerID="092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d" Jul 08 20:03:04.526858 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.526848 2573 scope.go:117] "RemoveContainer" containerID="6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1" Jul 08 20:03:04.533128 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.533112 2573 scope.go:117] "RemoveContainer" containerID="cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8" Jul 08 20:03:04.540265 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.540252 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 20:03:04.540337 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.540287 2573 scope.go:117] "RemoveContainer" containerID="ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440" Jul 08 20:03:04.546880 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.546859 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 20:03:04.547974 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.547959 2573 scope.go:117] "RemoveContainer" containerID="4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871" Jul 08 20:03:04.554450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.554436 2573 scope.go:117] "RemoveContainer" containerID="a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492" Jul 08 20:03:04.560898 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.560883 2573 scope.go:117] "RemoveContainer" containerID="6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf" Jul 08 20:03:04.561105 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:04.561089 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf\": container with ID starting with 6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf not found: ID does not exist" containerID="6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf" Jul 08 20:03:04.561150 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.561112 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf"} err="failed to get container status \"6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf\": rpc error: code = NotFound desc = could not find container \"6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf\": container with ID starting with 6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf not found: ID does not exist" Jul 08 20:03:04.561150 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.561128 2573 scope.go:117] "RemoveContainer" containerID="092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d" Jul 08 20:03:04.561380 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:04.561364 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d\": container with ID starting with 092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d not found: ID does not exist" containerID="092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d" Jul 08 20:03:04.561454 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.561388 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d"} err="failed to get container status \"092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d\": rpc error: code = NotFound desc = could not find container \"092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d\": container with ID starting with 092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d not found: ID does not exist" Jul 08 20:03:04.561454 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.561422 2573 scope.go:117] "RemoveContainer" containerID="6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1" Jul 08 20:03:04.561635 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:04.561620 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1\": container with ID starting with 6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1 not found: ID does not exist" containerID="6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1" Jul 08 20:03:04.561671 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.561638 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1"} err="failed to get container status \"6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1\": rpc error: code = NotFound desc = could not find container \"6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1\": container with ID starting with 6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1 not found: ID does not exist" Jul 08 20:03:04.561671 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.561650 2573 scope.go:117] "RemoveContainer" containerID="cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8" Jul 08 20:03:04.561838 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:04.561825 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8\": container with ID starting with cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8 not found: ID does not exist" containerID="cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8" Jul 08 20:03:04.561881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.561841 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8"} err="failed to get container status \"cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8\": rpc error: code = NotFound desc = could not find container \"cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8\": container with ID starting with cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8 not found: ID does not exist" Jul 08 20:03:04.561881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.561852 2573 scope.go:117] "RemoveContainer" containerID="ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440" Jul 08 20:03:04.562050 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:04.562035 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440\": container with ID starting with ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440 not found: ID does not exist" containerID="ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440" Jul 08 20:03:04.562085 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.562053 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440"} err="failed to get container status \"ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440\": rpc error: code = NotFound desc = could not find container \"ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440\": container with ID starting with ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440 not found: ID does not exist" Jul 08 20:03:04.562085 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.562065 2573 scope.go:117] "RemoveContainer" containerID="4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871" Jul 08 20:03:04.562267 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:04.562254 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871\": container with ID starting with 4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871 not found: ID does not exist" containerID="4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871" Jul 08 20:03:04.562303 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.562270 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871"} err="failed to get container status \"4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871\": rpc error: code = NotFound desc = could not find container \"4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871\": container with ID starting with 4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871 not found: ID does not exist" Jul 08 20:03:04.562303 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.562282 2573 scope.go:117] "RemoveContainer" containerID="a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492" Jul 08 20:03:04.562485 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:04.562467 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492\": container with ID starting with a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492 not found: ID does not exist" containerID="a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492" Jul 08 20:03:04.562525 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.562491 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492"} err="failed to get container status \"a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492\": rpc error: code = NotFound desc = could not find container \"a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492\": container with ID starting with a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492 not found: ID does not exist" Jul 08 20:03:04.562525 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.562507 2573 scope.go:117] "RemoveContainer" containerID="6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf" Jul 08 20:03:04.563103 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.562873 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf"} err="failed to get container status \"6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf\": rpc error: code = NotFound desc = could not find container \"6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf\": container with ID starting with 6977b5d03f9e46f9083ee0b0c18edbdfbf6f72b147cc1a583259feed0417bdbf not found: ID does not exist" Jul 08 20:03:04.563103 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.562892 2573 scope.go:117] "RemoveContainer" containerID="092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d" Jul 08 20:03:04.565180 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.565157 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d"} err="failed to get container status \"092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d\": rpc error: code = NotFound desc = could not find container \"092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d\": container with ID starting with 092ce76a6549fcf62b760aa61e1201613d116ac40a0599dcff5636f22d201f1d not found: ID does not exist" Jul 08 20:03:04.565180 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.565177 2573 scope.go:117] "RemoveContainer" containerID="6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1" Jul 08 20:03:04.565624 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.565603 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1"} err="failed to get container status \"6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1\": rpc error: code = NotFound desc = could not find container \"6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1\": container with ID starting with 6cd0193630ab5d6f28716c8768febae28309ba589c7513ceb91729ebf7ba97f1 not found: ID does not exist" Jul 08 20:03:04.565704 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.565626 2573 scope.go:117] "RemoveContainer" containerID="cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8" Jul 08 20:03:04.565834 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.565812 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8"} err="failed to get container status \"cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8\": rpc error: code = NotFound desc = could not find container \"cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8\": container with ID starting with cb87c4c332eb6267b1480b2a9d7b9aee9981306ee7bd13f80592ff22e93578f8 not found: ID does not exist" Jul 08 20:03:04.565834 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.565835 2573 scope.go:117] "RemoveContainer" containerID="ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440" Jul 08 20:03:04.566099 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.566074 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440"} err="failed to get container status \"ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440\": rpc error: code = NotFound desc = could not find container \"ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440\": container with ID starting with ebf3d5b65e40d7449b10852e77462ff0d8e6925c3d996a67f1d411f0cfd32440 not found: ID does not exist" Jul 08 20:03:04.566189 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.566101 2573 scope.go:117] "RemoveContainer" containerID="4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871" Jul 08 20:03:04.566427 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.566400 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871"} err="failed to get container status \"4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871\": rpc error: code = NotFound desc = could not find container \"4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871\": container with ID starting with 4f0b449756ad9fadb353e27a630def0c04ac1adf5f237f55e752c8025c740871 not found: ID does not exist" Jul 08 20:03:04.566544 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.566428 2573 scope.go:117] "RemoveContainer" containerID="a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492" Jul 08 20:03:04.566846 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.566821 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492"} err="failed to get container status \"a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492\": rpc error: code = NotFound desc = could not find container \"a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492\": container with ID starting with a33e98ec0541707e17cfcbbb56a016d18cc8dab38467c72df96f7c56987b2492 not found: ID does not exist" Jul 08 20:03:04.576989 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.576968 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 20:03:04.577252 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577239 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="alertmanager" Jul 08 20:03:04.577299 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577253 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="alertmanager" Jul 08 20:03:04.577299 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577260 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy" Jul 08 20:03:04.577299 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577266 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy" Jul 08 20:03:04.577299 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577274 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy-metric" Jul 08 20:03:04.577299 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577280 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy-metric" Jul 08 20:03:04.577299 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577287 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe1c02c-0aff-4c17-bec0-034d3c442963" containerName="console" Jul 08 20:03:04.577299 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577292 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe1c02c-0aff-4c17-bec0-034d3c442963" containerName="console" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577321 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="config-reloader" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577329 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="config-reloader" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577336 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="init-config-reloader" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577342 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="init-config-reloader" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577350 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy-web" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577355 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy-web" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577361 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="prom-label-proxy" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577366 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="prom-label-proxy" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577373 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61507dc-6151-4502-8b12-5d1604a83316" containerName="console" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577378 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61507dc-6151-4502-8b12-5d1604a83316" containerName="console" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577423 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577430 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="prom-label-proxy" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577436 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fe1c02c-0aff-4c17-bec0-034d3c442963" containerName="console" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577441 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy-metric" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577448 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="config-reloader" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577455 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="kube-rbac-proxy-web" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577461 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61507dc-6151-4502-8b12-5d1604a83316" containerName="console" Jul 08 20:03:04.577528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.577466 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" containerName="alertmanager" Jul 08 20:03:04.580876 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.580862 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.583803 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.583784 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 08 20:03:04.583803 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.583795 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 08 20:03:04.583935 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.583784 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 08 20:03:04.584045 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.584031 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 08 20:03:04.584115 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.584098 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 08 20:03:04.584155 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.584098 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 08 20:03:04.584155 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.584137 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4nzvq\"" Jul 08 20:03:04.584231 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.584149 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 08 20:03:04.584292 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.584274 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 08 20:03:04.590677 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.590661 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 08 20:03:04.601024 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.601006 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 20:03:04.673828 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.673801 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31b84820-0600-44c6-971a-1815fd067ed2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.673913 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.673841 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b84820-0600-44c6-971a-1815fd067ed2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.673913 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.673858 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vsw\" (UniqueName: \"kubernetes.io/projected/31b84820-0600-44c6-971a-1815fd067ed2-kube-api-access-r6vsw\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.673913 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.673884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-config-volume\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.674012 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.673911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.674012 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.673930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31b84820-0600-44c6-971a-1815fd067ed2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.674012 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.673969 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31b84820-0600-44c6-971a-1815fd067ed2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.674012 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.673997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31b84820-0600-44c6-971a-1815fd067ed2-config-out\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.674167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.674016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-web-config\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.674167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.674036 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.674167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.674068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.674167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.674084 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.674167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.674101 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775338 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775260 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b84820-0600-44c6-971a-1815fd067ed2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775338 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775298 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vsw\" (UniqueName: \"kubernetes.io/projected/31b84820-0600-44c6-971a-1815fd067ed2-kube-api-access-r6vsw\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775338 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-config-volume\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775401 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31b84820-0600-44c6-971a-1815fd067ed2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775440 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31b84820-0600-44c6-971a-1815fd067ed2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31b84820-0600-44c6-971a-1815fd067ed2-config-out\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775488 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-web-config\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775735 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775521 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775735 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775735 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775584 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775735 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775613 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775735 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775649 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31b84820-0600-44c6-971a-1815fd067ed2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.775965 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.775782 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/31b84820-0600-44c6-971a-1815fd067ed2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.776111 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.776084 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31b84820-0600-44c6-971a-1815fd067ed2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.776496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.776471 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31b84820-0600-44c6-971a-1815fd067ed2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.778626 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.778342 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/31b84820-0600-44c6-971a-1815fd067ed2-config-out\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.778626 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.778482 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-config-volume\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.778626 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.778488 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.778626 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.778583 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/31b84820-0600-44c6-971a-1815fd067ed2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.778810 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.778645 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-web-config\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.778810 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.778675 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.778881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.778813 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.778914 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.778888 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.780255 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.780237 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/31b84820-0600-44c6-971a-1815fd067ed2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.784442 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.784423 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vsw\" (UniqueName: \"kubernetes.io/projected/31b84820-0600-44c6-971a-1815fd067ed2-kube-api-access-r6vsw\") pod \"alertmanager-main-0\" (UID: \"31b84820-0600-44c6-971a-1815fd067ed2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:04.785243 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.785223 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70df7ff4-8608-4f65-842e-00a8a78c7f8c" path="/var/lib/kubelet/pods/70df7ff4-8608-4f65-842e-00a8a78c7f8c/volumes" Jul 08 20:03:04.890348 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:04.890330 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 08 20:03:05.062622 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:03:05.062597 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31b84820_0600_44c6_971a_1815fd067ed2.slice/crio-4dfa97de3aeaff2c5809c037c998e5ce069a0a09a3c3c2725f713cd463840843 WatchSource:0}: Error finding container 4dfa97de3aeaff2c5809c037c998e5ce069a0a09a3c3c2725f713cd463840843: Status 404 returned error can't find the container with id 4dfa97de3aeaff2c5809c037c998e5ce069a0a09a3c3c2725f713cd463840843 Jul 08 20:03:05.090540 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:05.090519 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 08 20:03:05.518454 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:05.518423 2573 generic.go:356] "Generic (PLEG): container finished" podID="31b84820-0600-44c6-971a-1815fd067ed2" containerID="b3b8932f4367312058677af9e55c8111ca990e1d147e27d67cd4cbf8008a70d4" exitCode=0 Jul 08 20:03:05.518594 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:05.518496 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31b84820-0600-44c6-971a-1815fd067ed2","Type":"ContainerDied","Data":"b3b8932f4367312058677af9e55c8111ca990e1d147e27d67cd4cbf8008a70d4"} Jul 08 20:03:05.518594 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:05.518566 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31b84820-0600-44c6-971a-1815fd067ed2","Type":"ContainerStarted","Data":"4dfa97de3aeaff2c5809c037c998e5ce069a0a09a3c3c2725f713cd463840843"} Jul 08 20:03:06.528085 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.528050 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31b84820-0600-44c6-971a-1815fd067ed2","Type":"ContainerStarted","Data":"ac4d98fe04812840e91c94cd01eb09521a114121ba41e69a0aabd556c75fa0de"} Jul 08 20:03:06.528557 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.528119 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31b84820-0600-44c6-971a-1815fd067ed2","Type":"ContainerStarted","Data":"34e9e21c2c1ab5a2a3a40205540b31414df498ee79c0e04901e1e4592adbdc4c"} Jul 08 20:03:06.528557 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.528149 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31b84820-0600-44c6-971a-1815fd067ed2","Type":"ContainerStarted","Data":"e5dfe85331ae78c00bffa12abbcd3c686de1415ff130de2d714bc8911a332f6c"} Jul 08 20:03:06.528557 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.528175 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31b84820-0600-44c6-971a-1815fd067ed2","Type":"ContainerStarted","Data":"c84d68b8afa2c0137cf8889383e992ac7b99f93cfaca1a92dacc3cbe96151f02"} Jul 08 20:03:06.528557 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.528201 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31b84820-0600-44c6-971a-1815fd067ed2","Type":"ContainerStarted","Data":"7e5c2aaf8582ee0e54c89534dda001f5e3db47d8a71aa2da069eb86c78498a91"} Jul 08 20:03:06.528557 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.528225 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"31b84820-0600-44c6-971a-1815fd067ed2","Type":"ContainerStarted","Data":"1b6bed2c2d3ac622a3081dfe59bc3a0d7cc36c9bdc59b5667c9e7ff23a1ef0d9"} Jul 08 20:03:06.559379 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.559327 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.544450982 podStartE2EDuration="2.559293178s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.842196ms" isStatefulPod=false podCreationTimestamp="2026-07-08 20:03:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:03:06.556872154 +0000 UTC m=+154.553889089" watchObservedRunningTime="2026-07-08 20:03:06.559293178 +0000 UTC m=+154.556310114" Jul 08 20:03:06.825615 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.824958 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p"] Jul 08 20:03:06.827662 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.827647 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.830643 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.830620 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 08 20:03:06.830752 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.830661 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jj5gq\"" Jul 08 20:03:06.830752 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.830686 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 08 20:03:06.830976 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.830954 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 08 20:03:06.830976 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.830970 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 08 20:03:06.831145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.830998 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 08 20:03:06.842238 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.840749 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 08 20:03:06.845766 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.845743 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p"] Jul 08 20:03:06.893493 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.893462 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.893493 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.893497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-federate-client-tls\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.893703 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.893582 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5407d5d5-b968-4216-ab1b-25dd81905065-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.893703 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.893611 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv7vn\" (UniqueName: \"kubernetes.io/projected/5407d5d5-b968-4216-ab1b-25dd81905065-kube-api-access-nv7vn\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.893703 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.893634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-secret-telemeter-client\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.893703 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.893682 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-telemeter-client-tls\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.893703 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.893704 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5407d5d5-b968-4216-ab1b-25dd81905065-serving-certs-ca-bundle\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.893895 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.893743 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5407d5d5-b968-4216-ab1b-25dd81905065-metrics-client-ca\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.995096 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.995066 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5407d5d5-b968-4216-ab1b-25dd81905065-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.995258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.995104 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv7vn\" (UniqueName: \"kubernetes.io/projected/5407d5d5-b968-4216-ab1b-25dd81905065-kube-api-access-nv7vn\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.995258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.995124 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-secret-telemeter-client\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.995258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.995157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-telemeter-client-tls\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.995258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.995184 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5407d5d5-b968-4216-ab1b-25dd81905065-serving-certs-ca-bundle\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.995258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.995206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5407d5d5-b968-4216-ab1b-25dd81905065-metrics-client-ca\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.995258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.995241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.995672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.995269 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-federate-client-tls\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.996092 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.996063 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5407d5d5-b968-4216-ab1b-25dd81905065-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.996092 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.996079 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5407d5d5-b968-4216-ab1b-25dd81905065-metrics-client-ca\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.996402 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.996381 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5407d5d5-b968-4216-ab1b-25dd81905065-serving-certs-ca-bundle\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.997797 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.997773 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.998145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.998127 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-telemeter-client-tls\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.998227 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.998172 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-federate-client-tls\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:06.998227 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:06.998189 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5407d5d5-b968-4216-ab1b-25dd81905065-secret-telemeter-client\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:07.004740 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.004715 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv7vn\" (UniqueName: \"kubernetes.io/projected/5407d5d5-b968-4216-ab1b-25dd81905065-kube-api-access-nv7vn\") pod \"telemeter-client-5bbf4dc7b9-frb7p\" (UID: \"5407d5d5-b968-4216-ab1b-25dd81905065\") " pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:07.128338 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.128236 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 20:03:07.128688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.128525 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="prometheus" containerID="cri-o://d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" gracePeriod=600 Jul 08 20:03:07.128688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.128554 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy" containerID="cri-o://7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" gracePeriod=600 Jul 08 20:03:07.128688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.128571 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy-thanos" containerID="cri-o://11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" gracePeriod=600 Jul 08 20:03:07.128688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.128573 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="thanos-sidecar" containerID="cri-o://9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" gracePeriod=600 Jul 08 20:03:07.128688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.128616 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="config-reloader" containerID="cri-o://534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" gracePeriod=600 Jul 08 20:03:07.129018 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.128752 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy-web" containerID="cri-o://d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" gracePeriod=600 Jul 08 20:03:07.141007 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.140825 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" Jul 08 20:03:07.302384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.302363 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p"] Jul 08 20:03:07.304395 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:03:07.304366 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5407d5d5_b968_4216_ab1b_25dd81905065.slice/crio-38a5bb52d35bdefef4eae84f6eb9659c3cb8fd288180b6363f5b31d92933323e WatchSource:0}: Error finding container 38a5bb52d35bdefef4eae84f6eb9659c3cb8fd288180b6363f5b31d92933323e: Status 404 returned error can't find the container with id 38a5bb52d35bdefef4eae84f6eb9659c3cb8fd288180b6363f5b31d92933323e Jul 08 20:03:07.384694 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.384627 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.398856 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.398830 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-0\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.398977 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.398863 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-kubelet-serving-ca-bundle\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.398977 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.398893 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config-out\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.398977 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.398965 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-kube-rbac-proxy\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.398996 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-2\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399025 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jzd2\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-kube-api-access-4jzd2\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399049 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399076 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399103 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-tls\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399151 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-grpc-tls\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399175 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-trusted-ca-bundle\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399193 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399214 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-1\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399244 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399280 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-metrics-client-certs\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399283 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399329 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-tls-assets\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399353 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-db\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399386 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-metrics-client-ca\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399411 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-serving-certs-ca-bundle\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399927 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399451 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-web-config\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399927 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399485 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-thanos-prometheus-http-client-file\") pod \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\" (UID: \"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99\") " Jul 08 20:03:07.399927 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399609 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:07.399927 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399783 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.399927 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399803 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.399927 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.399819 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-trusted-ca-bundle\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.400231 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.400205 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:07.400387 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.400360 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:07.401636 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.401609 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:07.401778 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.401754 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config-out" (OuterVolumeSpecName: "config-out") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:03:07.402016 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.401990 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config" (OuterVolumeSpecName: "config") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:07.402098 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.402044 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:07.402494 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.402391 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:03:07.402494 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.402466 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:07.403005 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.402894 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:07.403604 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.403574 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:07.403994 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.403969 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:07.404112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.404082 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:07.404588 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.404558 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:07.405170 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.405136 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-kube-api-access-4jzd2" (OuterVolumeSpecName: "kube-api-access-4jzd2") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "kube-api-access-4jzd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:03:07.405802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.405778 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:07.406574 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.406547 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:03:07.418984 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.418950 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-web-config" (OuterVolumeSpecName: "web-config") pod "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" (UID: "60f2b5db-c8a4-4c25-87db-f94cb6f6bd99"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:07.500993 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.500962 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-metrics-client-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.500993 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.500988 2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.500993 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501000 2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-web-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501008 2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-thanos-prometheus-http-client-file\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501019 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501027 2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config-out\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501035 2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-kube-rbac-proxy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501043 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jzd2\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-kube-api-access-4jzd2\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501051 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501060 2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501068 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501078 2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-grpc-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501086 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501094 2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501103 2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-secret-metrics-client-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501111 2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-tls-assets\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.501188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.501119 2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99-prometheus-k8s-db\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536362 2573 generic.go:356] "Generic (PLEG): container finished" podID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerID="11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" exitCode=0 Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536394 2573 generic.go:356] "Generic (PLEG): container finished" podID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerID="7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" exitCode=0 Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536403 2573 generic.go:356] "Generic (PLEG): container finished" podID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerID="d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" exitCode=0 Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536411 2573 generic.go:356] "Generic (PLEG): container finished" podID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerID="9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" exitCode=0 Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536419 2573 generic.go:356] "Generic (PLEG): container finished" podID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerID="534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" exitCode=0 Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536426 2573 generic.go:356] "Generic (PLEG): container finished" podID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerID="d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" exitCode=0 Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerDied","Data":"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e"} Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536585 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerDied","Data":"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6"} Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536623 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerDied","Data":"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079"} Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536659 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerDied","Data":"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734"} Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536693 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerDied","Data":"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba"} Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536731 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerDied","Data":"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590"} Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536767 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60f2b5db-c8a4-4c25-87db-f94cb6f6bd99","Type":"ContainerDied","Data":"0389cbc2753cece171e581c70ad47e8ca48102a7c95a45a105f5a242e94852b9"} Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.536810 2573 scope.go:117] "RemoveContainer" containerID="11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" Jul 08 20:03:07.538581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.537014 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.540780 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.540749 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" event={"ID":"5407d5d5-b968-4216-ab1b-25dd81905065","Type":"ContainerStarted","Data":"38a5bb52d35bdefef4eae84f6eb9659c3cb8fd288180b6363f5b31d92933323e"} Jul 08 20:03:07.546328 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.546292 2573 scope.go:117] "RemoveContainer" containerID="7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" Jul 08 20:03:07.553023 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.553010 2573 scope.go:117] "RemoveContainer" containerID="d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" Jul 08 20:03:07.559406 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.559386 2573 scope.go:117] "RemoveContainer" containerID="9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" Jul 08 20:03:07.566067 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.566044 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 20:03:07.567611 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.567588 2573 scope.go:117] "RemoveContainer" containerID="534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" Jul 08 20:03:07.571044 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.571025 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 20:03:07.574353 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.574334 2573 scope.go:117] "RemoveContainer" containerID="d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" Jul 08 20:03:07.580991 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.580976 2573 scope.go:117] "RemoveContainer" containerID="18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49" Jul 08 20:03:07.587245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.587229 2573 scope.go:117] "RemoveContainer" containerID="11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" Jul 08 20:03:07.587538 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:07.587514 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": container with ID starting with 11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e not found: ID does not exist" containerID="11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" Jul 08 20:03:07.587625 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.587545 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e"} err="failed to get container status \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": rpc error: code = NotFound desc = could not find container \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": container with ID starting with 11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e not found: ID does not exist" Jul 08 20:03:07.587625 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.587569 2573 scope.go:117] "RemoveContainer" containerID="7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" Jul 08 20:03:07.587819 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:07.587805 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": container with ID starting with 7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6 not found: ID does not exist" containerID="7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" Jul 08 20:03:07.587860 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.587825 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6"} err="failed to get container status \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": rpc error: code = NotFound desc = could not find container \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": container with ID starting with 7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6 not found: ID does not exist" Jul 08 20:03:07.587860 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.587838 2573 scope.go:117] "RemoveContainer" containerID="d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" Jul 08 20:03:07.588078 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:07.588065 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": container with ID starting with d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079 not found: ID does not exist" containerID="d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" Jul 08 20:03:07.588126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.588080 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079"} err="failed to get container status \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": rpc error: code = NotFound desc = could not find container \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": container with ID starting with d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079 not found: ID does not exist" Jul 08 20:03:07.588126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.588093 2573 scope.go:117] "RemoveContainer" containerID="9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" Jul 08 20:03:07.588325 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:07.588289 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": container with ID starting with 9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734 not found: ID does not exist" containerID="9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" Jul 08 20:03:07.588371 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.588329 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734"} err="failed to get container status \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": rpc error: code = NotFound desc = could not find container \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": container with ID starting with 9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734 not found: ID does not exist" Jul 08 20:03:07.588371 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.588344 2573 scope.go:117] "RemoveContainer" containerID="534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" Jul 08 20:03:07.588556 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:07.588542 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": container with ID starting with 534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba not found: ID does not exist" containerID="534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" Jul 08 20:03:07.588601 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.588558 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba"} err="failed to get container status \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": rpc error: code = NotFound desc = could not find container \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": container with ID starting with 534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba not found: ID does not exist" Jul 08 20:03:07.588601 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.588570 2573 scope.go:117] "RemoveContainer" containerID="d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" Jul 08 20:03:07.588807 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:07.588790 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": container with ID starting with d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590 not found: ID does not exist" containerID="d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" Jul 08 20:03:07.588861 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.588810 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590"} err="failed to get container status \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": rpc error: code = NotFound desc = could not find container \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": container with ID starting with d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590 not found: ID does not exist" Jul 08 20:03:07.588861 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.588836 2573 scope.go:117] "RemoveContainer" containerID="18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49" Jul 08 20:03:07.589027 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:07.589014 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": container with ID starting with 18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49 not found: ID does not exist" containerID="18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49" Jul 08 20:03:07.589060 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589029 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49"} err="failed to get container status \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": rpc error: code = NotFound desc = could not find container \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": container with ID starting with 18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49 not found: ID does not exist" Jul 08 20:03:07.589060 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589040 2573 scope.go:117] "RemoveContainer" containerID="11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" Jul 08 20:03:07.589232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589217 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e"} err="failed to get container status \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": rpc error: code = NotFound desc = could not find container \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": container with ID starting with 11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e not found: ID does not exist" Jul 08 20:03:07.589276 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589232 2573 scope.go:117] "RemoveContainer" containerID="7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" Jul 08 20:03:07.589455 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589435 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6"} err="failed to get container status \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": rpc error: code = NotFound desc = could not find container \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": container with ID starting with 7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6 not found: ID does not exist" Jul 08 20:03:07.589512 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589456 2573 scope.go:117] "RemoveContainer" containerID="d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" Jul 08 20:03:07.589677 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589658 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079"} err="failed to get container status \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": rpc error: code = NotFound desc = could not find container \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": container with ID starting with d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079 not found: ID does not exist" Jul 08 20:03:07.589734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589678 2573 scope.go:117] "RemoveContainer" containerID="9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" Jul 08 20:03:07.589880 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589861 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734"} err="failed to get container status \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": rpc error: code = NotFound desc = could not find container \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": container with ID starting with 9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734 not found: ID does not exist" Jul 08 20:03:07.589948 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.589880 2573 scope.go:117] "RemoveContainer" containerID="534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" Jul 08 20:03:07.590098 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590081 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba"} err="failed to get container status \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": rpc error: code = NotFound desc = could not find container \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": container with ID starting with 534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba not found: ID does not exist" Jul 08 20:03:07.590150 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590099 2573 scope.go:117] "RemoveContainer" containerID="d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" Jul 08 20:03:07.590331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590285 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590"} err="failed to get container status \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": rpc error: code = NotFound desc = could not find container \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": container with ID starting with d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590 not found: ID does not exist" Jul 08 20:03:07.590331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590328 2573 scope.go:117] "RemoveContainer" containerID="18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49" Jul 08 20:03:07.590561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590544 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49"} err="failed to get container status \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": rpc error: code = NotFound desc = could not find container \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": container with ID starting with 18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49 not found: ID does not exist" Jul 08 20:03:07.590612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590563 2573 scope.go:117] "RemoveContainer" containerID="11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" Jul 08 20:03:07.590764 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590748 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e"} err="failed to get container status \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": rpc error: code = NotFound desc = could not find container \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": container with ID starting with 11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e not found: ID does not exist" Jul 08 20:03:07.590804 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590765 2573 scope.go:117] "RemoveContainer" containerID="7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" Jul 08 20:03:07.590965 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590946 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6"} err="failed to get container status \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": rpc error: code = NotFound desc = could not find container \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": container with ID starting with 7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6 not found: ID does not exist" Jul 08 20:03:07.591007 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.590966 2573 scope.go:117] "RemoveContainer" containerID="d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" Jul 08 20:03:07.591168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591147 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079"} err="failed to get container status \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": rpc error: code = NotFound desc = could not find container \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": container with ID starting with d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079 not found: ID does not exist" Jul 08 20:03:07.591168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591168 2573 scope.go:117] "RemoveContainer" containerID="9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" Jul 08 20:03:07.591356 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591337 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734"} err="failed to get container status \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": rpc error: code = NotFound desc = could not find container \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": container with ID starting with 9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734 not found: ID does not exist" Jul 08 20:03:07.591356 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591354 2573 scope.go:117] "RemoveContainer" containerID="534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" Jul 08 20:03:07.591563 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591548 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba"} err="failed to get container status \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": rpc error: code = NotFound desc = could not find container \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": container with ID starting with 534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba not found: ID does not exist" Jul 08 20:03:07.591563 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591562 2573 scope.go:117] "RemoveContainer" containerID="d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" Jul 08 20:03:07.591735 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591721 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590"} err="failed to get container status \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": rpc error: code = NotFound desc = could not find container \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": container with ID starting with d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590 not found: ID does not exist" Jul 08 20:03:07.591776 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591734 2573 scope.go:117] "RemoveContainer" containerID="18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49" Jul 08 20:03:07.591908 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591889 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49"} err="failed to get container status \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": rpc error: code = NotFound desc = could not find container \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": container with ID starting with 18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49 not found: ID does not exist" Jul 08 20:03:07.591954 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.591910 2573 scope.go:117] "RemoveContainer" containerID="11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" Jul 08 20:03:07.592069 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592054 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e"} err="failed to get container status \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": rpc error: code = NotFound desc = could not find container \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": container with ID starting with 11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e not found: ID does not exist" Jul 08 20:03:07.592110 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592069 2573 scope.go:117] "RemoveContainer" containerID="7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" Jul 08 20:03:07.592245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592221 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6"} err="failed to get container status \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": rpc error: code = NotFound desc = could not find container \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": container with ID starting with 7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6 not found: ID does not exist" Jul 08 20:03:07.592283 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592246 2573 scope.go:117] "RemoveContainer" containerID="d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" Jul 08 20:03:07.592438 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592423 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079"} err="failed to get container status \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": rpc error: code = NotFound desc = could not find container \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": container with ID starting with d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079 not found: ID does not exist" Jul 08 20:03:07.592485 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592437 2573 scope.go:117] "RemoveContainer" containerID="9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" Jul 08 20:03:07.592609 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592593 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734"} err="failed to get container status \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": rpc error: code = NotFound desc = could not find container \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": container with ID starting with 9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734 not found: ID does not exist" Jul 08 20:03:07.592654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592608 2573 scope.go:117] "RemoveContainer" containerID="534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" Jul 08 20:03:07.592770 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592756 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba"} err="failed to get container status \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": rpc error: code = NotFound desc = could not find container \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": container with ID starting with 534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba not found: ID does not exist" Jul 08 20:03:07.592809 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592771 2573 scope.go:117] "RemoveContainer" containerID="d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" Jul 08 20:03:07.592928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592915 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590"} err="failed to get container status \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": rpc error: code = NotFound desc = could not find container \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": container with ID starting with d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590 not found: ID does not exist" Jul 08 20:03:07.592972 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.592927 2573 scope.go:117] "RemoveContainer" containerID="18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49" Jul 08 20:03:07.593127 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593113 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49"} err="failed to get container status \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": rpc error: code = NotFound desc = could not find container \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": container with ID starting with 18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49 not found: ID does not exist" Jul 08 20:03:07.593167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593127 2573 scope.go:117] "RemoveContainer" containerID="11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" Jul 08 20:03:07.593298 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593282 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e"} err="failed to get container status \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": rpc error: code = NotFound desc = could not find container \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": container with ID starting with 11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e not found: ID does not exist" Jul 08 20:03:07.593358 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593300 2573 scope.go:117] "RemoveContainer" containerID="7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" Jul 08 20:03:07.593539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593525 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6"} err="failed to get container status \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": rpc error: code = NotFound desc = could not find container \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": container with ID starting with 7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6 not found: ID does not exist" Jul 08 20:03:07.593539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593539 2573 scope.go:117] "RemoveContainer" containerID="d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" Jul 08 20:03:07.593708 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593687 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079"} err="failed to get container status \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": rpc error: code = NotFound desc = could not find container \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": container with ID starting with d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079 not found: ID does not exist" Jul 08 20:03:07.593755 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593710 2573 scope.go:117] "RemoveContainer" containerID="9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" Jul 08 20:03:07.593898 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593868 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734"} err="failed to get container status \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": rpc error: code = NotFound desc = could not find container \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": container with ID starting with 9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734 not found: ID does not exist" Jul 08 20:03:07.593936 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.593900 2573 scope.go:117] "RemoveContainer" containerID="534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" Jul 08 20:03:07.594057 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.594033 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba"} err="failed to get container status \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": rpc error: code = NotFound desc = could not find container \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": container with ID starting with 534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba not found: ID does not exist" Jul 08 20:03:07.594057 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.594049 2573 scope.go:117] "RemoveContainer" containerID="d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" Jul 08 20:03:07.594295 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.594264 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590"} err="failed to get container status \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": rpc error: code = NotFound desc = could not find container \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": container with ID starting with d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590 not found: ID does not exist" Jul 08 20:03:07.594295 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.594289 2573 scope.go:117] "RemoveContainer" containerID="18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49" Jul 08 20:03:07.594785 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.594742 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49"} err="failed to get container status \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": rpc error: code = NotFound desc = could not find container \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": container with ID starting with 18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49 not found: ID does not exist" Jul 08 20:03:07.594785 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.594768 2573 scope.go:117] "RemoveContainer" containerID="11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e" Jul 08 20:03:07.595159 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.595069 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e"} err="failed to get container status \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": rpc error: code = NotFound desc = could not find container \"11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e\": container with ID starting with 11d81a2ad9210fbf2acc1a06b1ca2b816de69ae5a32d45846cb33c9fcb1ebe8e not found: ID does not exist" Jul 08 20:03:07.595159 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.595095 2573 scope.go:117] "RemoveContainer" containerID="7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6" Jul 08 20:03:07.595388 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.595367 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6"} err="failed to get container status \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": rpc error: code = NotFound desc = could not find container \"7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6\": container with ID starting with 7d628d2b51ae9eda09be210bab5dc51ea0855ef19a8dc8e921b7badaa7abb4a6 not found: ID does not exist" Jul 08 20:03:07.595478 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.595388 2573 scope.go:117] "RemoveContainer" containerID="d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079" Jul 08 20:03:07.595710 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.595691 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079"} err="failed to get container status \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": rpc error: code = NotFound desc = could not find container \"d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079\": container with ID starting with d9758990b3edc42a4b754279fd75901903343446ae0240dc0608cd045d0cf079 not found: ID does not exist" Jul 08 20:03:07.595773 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.595711 2573 scope.go:117] "RemoveContainer" containerID="9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734" Jul 08 20:03:07.595963 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.595935 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734"} err="failed to get container status \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": rpc error: code = NotFound desc = could not find container \"9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734\": container with ID starting with 9704b90951a401aef277ee2e87058138563010f7593fa51b9f19f2a757394734 not found: ID does not exist" Jul 08 20:03:07.596019 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.595964 2573 scope.go:117] "RemoveContainer" containerID="534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba" Jul 08 20:03:07.596169 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596151 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 20:03:07.596223 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596164 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba"} err="failed to get container status \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": rpc error: code = NotFound desc = could not find container \"534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba\": container with ID starting with 534660a028d5cd838748f958df1f3c24f04151b3d1f0344f0391c9d4d99344ba not found: ID does not exist" Jul 08 20:03:07.596223 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596178 2573 scope.go:117] "RemoveContainer" containerID="d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590" Jul 08 20:03:07.596430 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596410 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590"} err="failed to get container status \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": rpc error: code = NotFound desc = could not find container \"d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590\": container with ID starting with d8f1e45f7c194064242b9efedc195b9d134c214f895822f274862bbb13028590 not found: ID does not exist" Jul 08 20:03:07.596430 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596429 2573 scope.go:117] "RemoveContainer" containerID="18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49" Jul 08 20:03:07.596600 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596583 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="prometheus" Jul 08 20:03:07.596651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596602 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="prometheus" Jul 08 20:03:07.596651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596616 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy-thanos" Jul 08 20:03:07.596651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596622 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy-thanos" Jul 08 20:03:07.596651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596630 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="config-reloader" Jul 08 20:03:07.596651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596636 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="config-reloader" Jul 08 20:03:07.596651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596631 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49"} err="failed to get container status \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": rpc error: code = NotFound desc = could not find container \"18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49\": container with ID starting with 18ddfc30d45860b483aaf782cb6d319e970765826386f85f87609e9bfab2cc49 not found: ID does not exist" Jul 08 20:03:07.596651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596642 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="thanos-sidecar" Jul 08 20:03:07.596651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596648 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="thanos-sidecar" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596658 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="init-config-reloader" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596663 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="init-config-reloader" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596671 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596675 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596683 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy-web" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596691 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy-web" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596740 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="prometheus" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596747 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="thanos-sidecar" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596755 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596767 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="config-reloader" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596774 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy-web" Jul 08 20:03:07.596879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.596779 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" containerName="kube-rbac-proxy-thanos" Jul 08 20:03:07.600393 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.600374 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.603126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603106 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 08 20:03:07.603212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603107 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 08 20:03:07.603212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603106 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-prldp\"" Jul 08 20:03:07.603360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603108 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 08 20:03:07.603360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603134 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 08 20:03:07.603360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603351 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 08 20:03:07.603513 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603444 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 08 20:03:07.603563 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603529 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 08 20:03:07.603608 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603577 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9su9qmk9e6k2v\"" Jul 08 20:03:07.603608 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.603586 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 08 20:03:07.604067 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.604054 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 08 20:03:07.604148 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.604055 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 08 20:03:07.604187 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.604058 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 08 20:03:07.604377 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.604351 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 08 20:03:07.607392 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.607374 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 08 20:03:07.609576 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.609561 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 08 20:03:07.617495 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.617476 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 20:03:07.702348 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-config\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702452 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702353 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702452 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702452 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702391 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702583 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702583 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702493 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702583 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702583 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702529 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702583 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17990734-0dd8-492b-8df8-f63a629f59e2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702741 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702606 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702741 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702624 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702741 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702655 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702741 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-web-config\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702741 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702741 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702926 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702747 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702926 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrqjb\" (UniqueName: \"kubernetes.io/projected/17990734-0dd8-492b-8df8-f63a629f59e2-kube-api-access-zrqjb\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702926 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702805 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17990734-0dd8-492b-8df8-f63a629f59e2-config-out\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702926 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.702926 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.702842 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.803865 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.803842 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17990734-0dd8-492b-8df8-f63a629f59e2-config-out\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.803951 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.803869 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.803951 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.803886 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.803951 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.803903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-config\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804066 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804066 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804066 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804059 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804150 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804181 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17990734-0dd8-492b-8df8-f63a629f59e2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804212 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804204 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804268 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804297 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-web-config\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804352 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804426 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804460 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrqjb\" (UniqueName: \"kubernetes.io/projected/17990734-0dd8-492b-8df8-f63a629f59e2-kube-api-access-zrqjb\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804490 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804997 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804626 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.804997 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.804663 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.805303 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.805280 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.806456 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.806424 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.806639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.806617 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17990734-0dd8-492b-8df8-f63a629f59e2-config-out\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.806774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.806755 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.807706 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.807297 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.807706 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.807304 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.807706 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.807374 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-config\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.807706 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.807380 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.807943 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.807830 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.808075 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.808047 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.808643 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.808614 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/17990734-0dd8-492b-8df8-f63a629f59e2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.808941 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.808917 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.809011 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.808988 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-web-config\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.809336 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.809298 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17990734-0dd8-492b-8df8-f63a629f59e2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.810009 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.809989 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.810065 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.810033 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/17990734-0dd8-492b-8df8-f63a629f59e2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.815116 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.815100 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrqjb\" (UniqueName: \"kubernetes.io/projected/17990734-0dd8-492b-8df8-f63a629f59e2-kube-api-access-zrqjb\") pod \"prometheus-k8s-0\" (UID: \"17990734-0dd8-492b-8df8-f63a629f59e2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:07.911101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:07.911074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:08.059643 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:08.059602 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 08 20:03:08.062427 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:03:08.062399 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17990734_0dd8_492b_8df8_f63a629f59e2.slice/crio-7d4ca41b9eeb15ec57b5e5cfbc7b464d836684a1d239efdca16392d5a520ab03 WatchSource:0}: Error finding container 7d4ca41b9eeb15ec57b5e5cfbc7b464d836684a1d239efdca16392d5a520ab03: Status 404 returned error can't find the container with id 7d4ca41b9eeb15ec57b5e5cfbc7b464d836684a1d239efdca16392d5a520ab03 Jul 08 20:03:08.549288 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:08.549259 2573 generic.go:356] "Generic (PLEG): container finished" podID="17990734-0dd8-492b-8df8-f63a629f59e2" containerID="829e8573e3b4a788de4ceadb5ea6c1968b145886424f37c404b02911bcb63818" exitCode=0 Jul 08 20:03:08.549629 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:08.549295 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17990734-0dd8-492b-8df8-f63a629f59e2","Type":"ContainerDied","Data":"829e8573e3b4a788de4ceadb5ea6c1968b145886424f37c404b02911bcb63818"} Jul 08 20:03:08.549629 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:08.549357 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17990734-0dd8-492b-8df8-f63a629f59e2","Type":"ContainerStarted","Data":"7d4ca41b9eeb15ec57b5e5cfbc7b464d836684a1d239efdca16392d5a520ab03"} Jul 08 20:03:08.790391 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:08.789947 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60f2b5db-c8a4-4c25-87db-f94cb6f6bd99" path="/var/lib/kubelet/pods/60f2b5db-c8a4-4c25-87db-f94cb6f6bd99/volumes" Jul 08 20:03:09.561347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:09.561138 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17990734-0dd8-492b-8df8-f63a629f59e2","Type":"ContainerStarted","Data":"76149b209a1926271aa65a066e2fcd73098b042fdbc8eadabd196ef977fb8ce3"} Jul 08 20:03:09.561347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:09.561217 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17990734-0dd8-492b-8df8-f63a629f59e2","Type":"ContainerStarted","Data":"97db7babcba15c7bdc1a4fdb303496c851e28b896d40ad8adb3db422bb5e7bad"} Jul 08 20:03:09.561347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:09.561256 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17990734-0dd8-492b-8df8-f63a629f59e2","Type":"ContainerStarted","Data":"804e4fe0c2fbf4dadb3bc17a7b26c8a8fd358f02c259c8654ca1d2299a84ad37"} Jul 08 20:03:09.561347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:09.561293 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17990734-0dd8-492b-8df8-f63a629f59e2","Type":"ContainerStarted","Data":"7fd193b5d4881dc3e4b6e1ead7b526e2c49a93719ecda2269a689e1d155748cf"} Jul 08 20:03:09.561347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:09.561347 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17990734-0dd8-492b-8df8-f63a629f59e2","Type":"ContainerStarted","Data":"f12dcef4d1aaf595114dd1284ad25b05b0dd7af59f0265f1be6de6b1a653b50f"} Jul 08 20:03:09.561790 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:09.561385 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"17990734-0dd8-492b-8df8-f63a629f59e2","Type":"ContainerStarted","Data":"4d462135c9cff88a53ea121ccb88b86ab3dd6c232504deab43c2954d7ab2f8a0"} Jul 08 20:03:09.593324 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:09.593132 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5711330009999998 podStartE2EDuration="2.593108965s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.975964ms" isStatefulPod=false podCreationTimestamp="2026-07-08 20:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:03:09.591222646 +0000 UTC m=+157.588239582" watchObservedRunningTime="2026-07-08 20:03:09.593108965 +0000 UTC m=+157.590125901" Jul 08 20:03:10.567136 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:10.567093 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" event={"ID":"5407d5d5-b968-4216-ab1b-25dd81905065","Type":"ContainerStarted","Data":"bc7efe44f6b6d5248c9487832883b59dbc70f0dbd37e48a7c6f9629c745aaea1"} Jul 08 20:03:10.567528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:10.567195 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" event={"ID":"5407d5d5-b968-4216-ab1b-25dd81905065","Type":"ContainerStarted","Data":"b4f0d0bc5ec0bf561e276b5eab908693e5363afb86407e7a2737177024c7d091"} Jul 08 20:03:10.567528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:10.567229 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" event={"ID":"5407d5d5-b968-4216-ab1b-25dd81905065","Type":"ContainerStarted","Data":"2c503f7a882fc2434e11edfecf856d79271e6bf442a5e4b138a84a7f1ad1245f"} Jul 08 20:03:10.591561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:10.591513 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bbf4dc7b9-frb7p" podStartSLOduration=2.3576265149999998 podStartE2EDuration="4.591499896s" totalImagesPullingTime="2.233873381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:03:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:03:10.590539824 +0000 UTC m=+158.587556760" watchObservedRunningTime="2026-07-08 20:03:10.591499896 +0000 UTC m=+158.588516825" Jul 08 20:03:11.414937 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.414904 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-595678b7c5-9v26f"] Jul 08 20:03:11.421110 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.421084 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.430478 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.430452 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595678b7c5-9v26f"] Jul 08 20:03:11.481548 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.481517 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:11.538100 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.538068 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-service-ca\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.538236 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.538106 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qblf5\" (UniqueName: \"kubernetes.io/projected/2299183c-ad59-4a95-989b-f7d09623bf7c-kube-api-access-qblf5\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.538236 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.538136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-serving-cert\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.538298 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.538254 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-console-config\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.538298 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.538287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-oauth-config\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.538413 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.538332 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-oauth-serving-cert\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.538413 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.538380 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-trusted-ca-bundle\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.549045 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.549018 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:11.639079 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.639051 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-console-config\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.639533 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.639093 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-oauth-config\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.639533 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.639125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-oauth-serving-cert\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.639533 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.639159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-trusted-ca-bundle\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.639533 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.639364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-service-ca\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.639533 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.639395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qblf5\" (UniqueName: \"kubernetes.io/projected/2299183c-ad59-4a95-989b-f7d09623bf7c-kube-api-access-qblf5\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.639533 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.639416 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-serving-cert\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.639947 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.639926 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-oauth-serving-cert\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.640129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.640105 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-service-ca\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.640178 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.640133 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-console-config\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.640178 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.640154 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-trusted-ca-bundle\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.641774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.641751 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-serving-cert\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.642033 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.642017 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-oauth-config\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.648527 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.648503 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qblf5\" (UniqueName: \"kubernetes.io/projected/2299183c-ad59-4a95-989b-f7d09623bf7c-kube-api-access-qblf5\") pod \"console-595678b7c5-9v26f\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.730587 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.730540 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:11.869933 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:11.869903 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-595678b7c5-9v26f"] Jul 08 20:03:11.871892 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:03:11.871864 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2299183c_ad59_4a95_989b_f7d09623bf7c.slice/crio-106db60e22d17caf9d932ca00987bce840eb7655d9449b9e8cd3a0e731b6d72b WatchSource:0}: Error finding container 106db60e22d17caf9d932ca00987bce840eb7655d9449b9e8cd3a0e731b6d72b: Status 404 returned error can't find the container with id 106db60e22d17caf9d932ca00987bce840eb7655d9449b9e8cd3a0e731b6d72b Jul 08 20:03:12.577593 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:12.577561 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595678b7c5-9v26f" event={"ID":"2299183c-ad59-4a95-989b-f7d09623bf7c","Type":"ContainerStarted","Data":"8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1"} Jul 08 20:03:12.577762 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:12.577607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595678b7c5-9v26f" event={"ID":"2299183c-ad59-4a95-989b-f7d09623bf7c","Type":"ContainerStarted","Data":"106db60e22d17caf9d932ca00987bce840eb7655d9449b9e8cd3a0e731b6d72b"} Jul 08 20:03:12.603424 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:12.601071 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-595678b7c5-9v26f" podStartSLOduration=1.601054037 podStartE2EDuration="1.601054037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:03:12.599606261 +0000 UTC m=+160.596623197" watchObservedRunningTime="2026-07-08 20:03:12.601054037 +0000 UTC m=+160.598070977" Jul 08 20:03:12.911469 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:12.911392 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:03:21.730819 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:21.730769 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:21.731301 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:21.730879 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:21.735727 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:21.735705 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:22.622291 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:22.622260 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:03:22.675617 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:22.675585 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cd786c966-2s9w8"] Jul 08 20:03:22.809388 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:22.809352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:22.910484 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:22.910408 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:22.971703 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:22.971674 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:33.725547 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:33.725514 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:33.781963 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:33.781931 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:03:47.700557 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.700489 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7cd786c966-2s9w8" podUID="250580a7-f77b-4f07-a286-231398b0a786" containerName="console" containerID="cri-o://ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55" gracePeriod=15 Jul 08 20:03:47.941997 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.941968 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:03:47.973835 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.973754 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-service-ca\") pod \"250580a7-f77b-4f07-a286-231398b0a786\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " Jul 08 20:03:47.973835 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.973809 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l8g7\" (UniqueName: \"kubernetes.io/projected/250580a7-f77b-4f07-a286-231398b0a786-kube-api-access-2l8g7\") pod \"250580a7-f77b-4f07-a286-231398b0a786\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " Jul 08 20:03:47.974050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.973869 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-serving-cert\") pod \"250580a7-f77b-4f07-a286-231398b0a786\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " Jul 08 20:03:47.974050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.973894 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-oauth-config\") pod \"250580a7-f77b-4f07-a286-231398b0a786\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " Jul 08 20:03:47.974050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.973930 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-oauth-serving-cert\") pod \"250580a7-f77b-4f07-a286-231398b0a786\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " Jul 08 20:03:47.974050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.974032 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-console-config\") pod \"250580a7-f77b-4f07-a286-231398b0a786\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " Jul 08 20:03:47.974245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.974057 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-trusted-ca-bundle\") pod \"250580a7-f77b-4f07-a286-231398b0a786\" (UID: \"250580a7-f77b-4f07-a286-231398b0a786\") " Jul 08 20:03:47.974245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.974149 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-service-ca" (OuterVolumeSpecName: "service-ca") pod "250580a7-f77b-4f07-a286-231398b0a786" (UID: "250580a7-f77b-4f07-a286-231398b0a786"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:47.974465 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.974421 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "250580a7-f77b-4f07-a286-231398b0a786" (UID: "250580a7-f77b-4f07-a286-231398b0a786"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:47.974465 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.974454 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:47.974683 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.974507 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "250580a7-f77b-4f07-a286-231398b0a786" (UID: "250580a7-f77b-4f07-a286-231398b0a786"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:47.974749 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.974711 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-console-config" (OuterVolumeSpecName: "console-config") pod "250580a7-f77b-4f07-a286-231398b0a786" (UID: "250580a7-f77b-4f07-a286-231398b0a786"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:03:47.976114 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.976087 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "250580a7-f77b-4f07-a286-231398b0a786" (UID: "250580a7-f77b-4f07-a286-231398b0a786"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:47.976114 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.976102 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "250580a7-f77b-4f07-a286-231398b0a786" (UID: "250580a7-f77b-4f07-a286-231398b0a786"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:03:47.976114 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:47.976097 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250580a7-f77b-4f07-a286-231398b0a786-kube-api-access-2l8g7" (OuterVolumeSpecName: "kube-api-access-2l8g7") pod "250580a7-f77b-4f07-a286-231398b0a786" (UID: "250580a7-f77b-4f07-a286-231398b0a786"). InnerVolumeSpecName "kube-api-access-2l8g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:03:48.075966 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.075921 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:48.075966 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.075959 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/250580a7-f77b-4f07-a286-231398b0a786-console-oauth-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:48.075966 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.075968 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-oauth-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:48.075966 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.075978 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-console-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:48.076239 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.075986 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/250580a7-f77b-4f07-a286-231398b0a786-trusted-ca-bundle\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:48.076239 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.075995 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l8g7\" (UniqueName: \"kubernetes.io/projected/250580a7-f77b-4f07-a286-231398b0a786-kube-api-access-2l8g7\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:03:48.721653 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.721614 2573 generic.go:356] "Generic (PLEG): container finished" podID="250580a7-f77b-4f07-a286-231398b0a786" containerID="ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55" exitCode=0 Jul 08 20:03:48.722113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.721669 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cd786c966-2s9w8" Jul 08 20:03:48.722113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.721670 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd786c966-2s9w8" event={"ID":"250580a7-f77b-4f07-a286-231398b0a786","Type":"ContainerDied","Data":"ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55"} Jul 08 20:03:48.722113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.721791 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cd786c966-2s9w8" event={"ID":"250580a7-f77b-4f07-a286-231398b0a786","Type":"ContainerDied","Data":"f0b6a0e8632791489633671b3fb3fed45011a4a53fd3dec7074f6c1331023b66"} Jul 08 20:03:48.722113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.721815 2573 scope.go:117] "RemoveContainer" containerID="ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55" Jul 08 20:03:48.730218 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.730198 2573 scope.go:117] "RemoveContainer" containerID="ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55" Jul 08 20:03:48.730539 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:03:48.730518 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55\": container with ID starting with ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55 not found: ID does not exist" containerID="ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55" Jul 08 20:03:48.730635 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.730546 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55"} err="failed to get container status \"ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55\": rpc error: code = NotFound desc = could not find container \"ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55\": container with ID starting with ff9eb3ea44a6a56eaac4f17b3d1d0d57934b601c2dc78cc93ca4468c11d5ac55 not found: ID does not exist" Jul 08 20:03:48.744860 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.744827 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7cd786c966-2s9w8"] Jul 08 20:03:48.751555 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.751518 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7cd786c966-2s9w8"] Jul 08 20:03:48.785949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:48.785915 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250580a7-f77b-4f07-a286-231398b0a786" path="/var/lib/kubelet/pods/250580a7-f77b-4f07-a286-231398b0a786/volumes" Jul 08 20:03:49.535435 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:03:49.535388 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:07.911696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:07.911666 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:04:07.926359 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:07.926339 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:04:08.822170 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:08.822141 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 08 20:04:16.978980 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:16.978941 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:17.091597 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.091566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:17.135474 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.135434 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-597557c7d4-5fgck"] Jul 08 20:04:17.135848 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.135830 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250580a7-f77b-4f07-a286-231398b0a786" containerName="console" Jul 08 20:04:17.135949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.135850 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="250580a7-f77b-4f07-a286-231398b0a786" containerName="console" Jul 08 20:04:17.136004 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.135970 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="250580a7-f77b-4f07-a286-231398b0a786" containerName="console" Jul 08 20:04:17.140330 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.140296 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.153204 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.153177 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597557c7d4-5fgck"] Jul 08 20:04:17.215973 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.215943 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:17.241457 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.241364 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-oauth-config\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.241457 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.241400 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-oauth-serving-cert\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.241457 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.241418 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57z6s\" (UniqueName: \"kubernetes.io/projected/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-kube-api-access-57z6s\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.241671 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.241497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-serving-cert\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.241671 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.241518 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-trusted-ca-bundle\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.241671 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.241562 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-service-ca\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.241671 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.241604 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-config\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.294474 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.294434 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:17.342567 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.342533 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-trusted-ca-bundle\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.342726 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.342614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-service-ca\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.342726 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.342658 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-config\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.342726 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.342705 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-oauth-config\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.342884 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.342727 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-oauth-serving-cert\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.342884 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.342750 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57z6s\" (UniqueName: \"kubernetes.io/projected/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-kube-api-access-57z6s\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.342884 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.342831 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-serving-cert\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.343530 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.343502 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-config\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.343677 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.343590 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-oauth-serving-cert\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.343677 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.343605 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-trusted-ca-bundle\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.343677 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.343638 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-service-ca\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.345275 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.345257 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-serving-cert\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.345546 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.345528 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-oauth-config\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.352537 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.352507 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57z6s\" (UniqueName: \"kubernetes.io/projected/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-kube-api-access-57z6s\") pod \"console-597557c7d4-5fgck\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.449992 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.449953 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:17.586294 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.586265 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597557c7d4-5fgck"] Jul 08 20:04:17.588814 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:04:17.588786 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beb52b3_cda5_48a1_a109_56b9e5afbb6f.slice/crio-28980baa3bc5d8074cde4ac4ad5dc9b11e14ad28774bd48568d6e453a00964d2 WatchSource:0}: Error finding container 28980baa3bc5d8074cde4ac4ad5dc9b11e14ad28774bd48568d6e453a00964d2: Status 404 returned error can't find the container with id 28980baa3bc5d8074cde4ac4ad5dc9b11e14ad28774bd48568d6e453a00964d2 Jul 08 20:04:17.843368 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.843251 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597557c7d4-5fgck" event={"ID":"3beb52b3-cda5-48a1-a109-56b9e5afbb6f","Type":"ContainerStarted","Data":"b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b"} Jul 08 20:04:17.843368 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.843356 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597557c7d4-5fgck" event={"ID":"3beb52b3-cda5-48a1-a109-56b9e5afbb6f","Type":"ContainerStarted","Data":"28980baa3bc5d8074cde4ac4ad5dc9b11e14ad28774bd48568d6e453a00964d2"} Jul 08 20:04:17.864718 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:17.864651 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-597557c7d4-5fgck" podStartSLOduration=0.864634435 podStartE2EDuration="864.634435ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:04:17.863948306 +0000 UTC m=+225.860965266" watchObservedRunningTime="2026-07-08 20:04:17.864634435 +0000 UTC m=+225.861651371" Jul 08 20:04:27.450964 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:27.450916 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:27.451551 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:27.450987 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:27.455821 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:27.455802 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:27.891510 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:27.891426 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:04:27.967792 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:27.967759 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595678b7c5-9v26f"] Jul 08 20:04:28.143012 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:28.142900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:28.240896 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:28.240829 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:28.344152 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:28.344120 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:29.023916 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:29.023885 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:29.129796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:29.129756 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:36.559554 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:36.559518 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:39.468890 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:39.468850 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:39.567959 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:39.567929 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:04:52.987988 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:52.987946 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-595678b7c5-9v26f" podUID="2299183c-ad59-4a95-989b-f7d09623bf7c" containerName="console" containerID="cri-o://8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1" gracePeriod=15 Jul 08 20:04:53.236575 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.236548 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:04:53.350453 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.350383 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qblf5\" (UniqueName: \"kubernetes.io/projected/2299183c-ad59-4a95-989b-f7d09623bf7c-kube-api-access-qblf5\") pod \"2299183c-ad59-4a95-989b-f7d09623bf7c\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " Jul 08 20:04:53.350453 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.350414 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-serving-cert\") pod \"2299183c-ad59-4a95-989b-f7d09623bf7c\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " Jul 08 20:04:53.350453 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.350432 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-oauth-config\") pod \"2299183c-ad59-4a95-989b-f7d09623bf7c\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " Jul 08 20:04:53.350713 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.350459 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-oauth-serving-cert\") pod \"2299183c-ad59-4a95-989b-f7d09623bf7c\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " Jul 08 20:04:53.350713 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.350582 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-console-config\") pod \"2299183c-ad59-4a95-989b-f7d09623bf7c\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " Jul 08 20:04:53.350713 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.350657 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-service-ca\") pod \"2299183c-ad59-4a95-989b-f7d09623bf7c\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " Jul 08 20:04:53.350713 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.350675 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-trusted-ca-bundle\") pod \"2299183c-ad59-4a95-989b-f7d09623bf7c\" (UID: \"2299183c-ad59-4a95-989b-f7d09623bf7c\") " Jul 08 20:04:53.350923 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.350846 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2299183c-ad59-4a95-989b-f7d09623bf7c" (UID: "2299183c-ad59-4a95-989b-f7d09623bf7c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:04:53.350980 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.350961 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-console-config" (OuterVolumeSpecName: "console-config") pod "2299183c-ad59-4a95-989b-f7d09623bf7c" (UID: "2299183c-ad59-4a95-989b-f7d09623bf7c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:04:53.351058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.351040 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-oauth-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:04:53.351114 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.351062 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-console-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:04:53.351114 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.351056 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-service-ca" (OuterVolumeSpecName: "service-ca") pod "2299183c-ad59-4a95-989b-f7d09623bf7c" (UID: "2299183c-ad59-4a95-989b-f7d09623bf7c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:04:53.351217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.351165 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2299183c-ad59-4a95-989b-f7d09623bf7c" (UID: "2299183c-ad59-4a95-989b-f7d09623bf7c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:04:53.352620 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.352594 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2299183c-ad59-4a95-989b-f7d09623bf7c" (UID: "2299183c-ad59-4a95-989b-f7d09623bf7c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:04:53.352972 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.352953 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2299183c-ad59-4a95-989b-f7d09623bf7c" (UID: "2299183c-ad59-4a95-989b-f7d09623bf7c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:04:53.353045 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.352957 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2299183c-ad59-4a95-989b-f7d09623bf7c-kube-api-access-qblf5" (OuterVolumeSpecName: "kube-api-access-qblf5") pod "2299183c-ad59-4a95-989b-f7d09623bf7c" (UID: "2299183c-ad59-4a95-989b-f7d09623bf7c"). InnerVolumeSpecName "kube-api-access-qblf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:04:53.452188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.452154 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qblf5\" (UniqueName: \"kubernetes.io/projected/2299183c-ad59-4a95-989b-f7d09623bf7c-kube-api-access-qblf5\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:04:53.452188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.452186 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:04:53.452188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.452195 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2299183c-ad59-4a95-989b-f7d09623bf7c-console-oauth-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:04:53.452396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.452203 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:04:53.452396 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.452212 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2299183c-ad59-4a95-989b-f7d09623bf7c-trusted-ca-bundle\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:04:53.996112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.996077 2573 generic.go:356] "Generic (PLEG): container finished" podID="2299183c-ad59-4a95-989b-f7d09623bf7c" containerID="8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1" exitCode=0 Jul 08 20:04:53.996491 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.996120 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-595678b7c5-9v26f" Jul 08 20:04:53.996491 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.996158 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595678b7c5-9v26f" event={"ID":"2299183c-ad59-4a95-989b-f7d09623bf7c","Type":"ContainerDied","Data":"8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1"} Jul 08 20:04:53.996491 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.996213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-595678b7c5-9v26f" event={"ID":"2299183c-ad59-4a95-989b-f7d09623bf7c","Type":"ContainerDied","Data":"106db60e22d17caf9d932ca00987bce840eb7655d9449b9e8cd3a0e731b6d72b"} Jul 08 20:04:53.996491 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:53.996235 2573 scope.go:117] "RemoveContainer" containerID="8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1" Jul 08 20:04:54.004346 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:54.004332 2573 scope.go:117] "RemoveContainer" containerID="8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1" Jul 08 20:04:54.004623 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:04:54.004607 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1\": container with ID starting with 8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1 not found: ID does not exist" containerID="8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1" Jul 08 20:04:54.004685 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:54.004633 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1"} err="failed to get container status \"8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1\": rpc error: code = NotFound desc = could not find container \"8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1\": container with ID starting with 8b75800a2e81d4b9942e6f7cbc2219f6d08883ef757e3e6d89c9a9d446777ce1 not found: ID does not exist" Jul 08 20:04:54.018809 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:54.018789 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-595678b7c5-9v26f"] Jul 08 20:04:54.023934 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:54.023916 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-595678b7c5-9v26f"] Jul 08 20:04:54.785760 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:04:54.785731 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2299183c-ad59-4a95-989b-f7d09623bf7c" path="/var/lib/kubelet/pods/2299183c-ad59-4a95-989b-f7d09623bf7c/volumes" Jul 08 20:05:13.406913 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:13.406867 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:13.520932 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:13.520906 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:32.522391 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:32.522360 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:05:32.524922 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:32.524901 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:05:32.528378 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:32.528361 2573 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 20:05:40.858675 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:40.858644 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:42.912591 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:42.912559 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:45.086986 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:45.086946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:45.188065 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:45.188034 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:45.949904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:45.949876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:46.082332 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:46.082285 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:46.806688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:46.806663 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:46.918186 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:46.918159 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:49.165997 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.165965 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:49.390335 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.390287 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:49.532540 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.532467 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:49.629895 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.629870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:49.693622 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.693599 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q"] Jul 08 20:05:49.693994 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.693976 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2299183c-ad59-4a95-989b-f7d09623bf7c" containerName="console" Jul 08 20:05:49.694072 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.693997 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2299183c-ad59-4a95-989b-f7d09623bf7c" containerName="console" Jul 08 20:05:49.694134 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.694126 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2299183c-ad59-4a95-989b-f7d09623bf7c" containerName="console" Jul 08 20:05:49.697146 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.697126 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" Jul 08 20:05:49.700219 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.700200 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 20:05:49.701388 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.701372 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jgvb5\"" Jul 08 20:05:49.701571 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.701425 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 20:05:49.711299 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.711277 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q"] Jul 08 20:05:49.796913 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.796867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7l9n\" (UniqueName: \"kubernetes.io/projected/19a1afcc-e7f6-4735-846a-c7396bd61527-kube-api-access-j7l9n\") pod \"cert-manager-cainjector-559cdd8c87-5zh2q\" (UID: \"19a1afcc-e7f6-4735-846a-c7396bd61527\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" Jul 08 20:05:49.796913 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.796903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19a1afcc-e7f6-4735-846a-c7396bd61527-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5zh2q\" (UID: \"19a1afcc-e7f6-4735-846a-c7396bd61527\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" Jul 08 20:05:49.897144 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.897125 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19a1afcc-e7f6-4735-846a-c7396bd61527-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5zh2q\" (UID: \"19a1afcc-e7f6-4735-846a-c7396bd61527\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" Jul 08 20:05:49.897237 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.897208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7l9n\" (UniqueName: \"kubernetes.io/projected/19a1afcc-e7f6-4735-846a-c7396bd61527-kube-api-access-j7l9n\") pod \"cert-manager-cainjector-559cdd8c87-5zh2q\" (UID: \"19a1afcc-e7f6-4735-846a-c7396bd61527\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" Jul 08 20:05:49.907249 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.907230 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/19a1afcc-e7f6-4735-846a-c7396bd61527-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5zh2q\" (UID: \"19a1afcc-e7f6-4735-846a-c7396bd61527\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" Jul 08 20:05:49.907504 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:49.907484 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7l9n\" (UniqueName: \"kubernetes.io/projected/19a1afcc-e7f6-4735-846a-c7396bd61527-kube-api-access-j7l9n\") pod \"cert-manager-cainjector-559cdd8c87-5zh2q\" (UID: \"19a1afcc-e7f6-4735-846a-c7396bd61527\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" Jul 08 20:05:50.016143 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:50.016119 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" Jul 08 20:05:50.146220 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:50.146198 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q"] Jul 08 20:05:50.148672 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:05:50.148648 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19a1afcc_e7f6_4735_846a_c7396bd61527.slice/crio-e784a829eec338d04b25b33bac2cd648184e721cab6565c9d8658c1e6892114b WatchSource:0}: Error finding container e784a829eec338d04b25b33bac2cd648184e721cab6565c9d8658c1e6892114b: Status 404 returned error can't find the container with id e784a829eec338d04b25b33bac2cd648184e721cab6565c9d8658c1e6892114b Jul 08 20:05:50.150934 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:50.150920 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:05:50.227945 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:50.227920 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" event={"ID":"19a1afcc-e7f6-4735-846a-c7396bd61527","Type":"ContainerStarted","Data":"e784a829eec338d04b25b33bac2cd648184e721cab6565c9d8658c1e6892114b"} Jul 08 20:05:51.878799 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:51.878763 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:51.973504 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:51.973444 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:53.244631 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:53.244607 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" event={"ID":"19a1afcc-e7f6-4735-846a-c7396bd61527","Type":"ContainerStarted","Data":"104067e6a709d4c10ea33a3aa122a8691dc8eba0df9ed561f85d64641924fc28"} Jul 08 20:05:53.280742 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:53.280699 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5zh2q" podStartSLOduration=1.375885891 podStartE2EDuration="4.280686052s" totalImagesPullingTime="2.904800161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:05:53.280654871 +0000 UTC m=+321.277671806" watchObservedRunningTime="2026-07-08 20:05:53.280686052 +0000 UTC m=+321.277702986" Jul 08 20:05:56.803575 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:56.803543 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:58.791697 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:58.791653 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:58.943710 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:58.943677 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:05:59.982460 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:59.982429 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv"] Jul 08 20:05:59.985292 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:59.985267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" Jul 08 20:05:59.988024 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:59.987993 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 20:05:59.989066 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:59.989050 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-v8vmk\"" Jul 08 20:05:59.989166 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:59.989128 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 20:05:59.998864 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:05:59.998847 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv"] Jul 08 20:06:00.083450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:00.083427 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4bv7\" (UniqueName: \"kubernetes.io/projected/865977f3-fd8b-45f3-867c-951902a420ea-kube-api-access-l4bv7\") pod \"openshift-lws-operator-fd8ccff4c-79bfv\" (UID: \"865977f3-fd8b-45f3-867c-951902a420ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" Jul 08 20:06:00.083545 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:00.083471 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/865977f3-fd8b-45f3-867c-951902a420ea-tmp\") pod \"openshift-lws-operator-fd8ccff4c-79bfv\" (UID: \"865977f3-fd8b-45f3-867c-951902a420ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" Jul 08 20:06:00.184106 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:00.184083 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4bv7\" (UniqueName: \"kubernetes.io/projected/865977f3-fd8b-45f3-867c-951902a420ea-kube-api-access-l4bv7\") pod \"openshift-lws-operator-fd8ccff4c-79bfv\" (UID: \"865977f3-fd8b-45f3-867c-951902a420ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" Jul 08 20:06:00.184191 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:00.184129 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/865977f3-fd8b-45f3-867c-951902a420ea-tmp\") pod \"openshift-lws-operator-fd8ccff4c-79bfv\" (UID: \"865977f3-fd8b-45f3-867c-951902a420ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" Jul 08 20:06:00.184502 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:00.184476 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/865977f3-fd8b-45f3-867c-951902a420ea-tmp\") pod \"openshift-lws-operator-fd8ccff4c-79bfv\" (UID: \"865977f3-fd8b-45f3-867c-951902a420ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" Jul 08 20:06:00.193819 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:00.193797 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4bv7\" (UniqueName: \"kubernetes.io/projected/865977f3-fd8b-45f3-867c-951902a420ea-kube-api-access-l4bv7\") pod \"openshift-lws-operator-fd8ccff4c-79bfv\" (UID: \"865977f3-fd8b-45f3-867c-951902a420ea\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" Jul 08 20:06:00.294394 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:00.294337 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" Jul 08 20:06:00.427323 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:00.427285 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv"] Jul 08 20:06:00.431246 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:06:00.431221 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod865977f3_fd8b_45f3_867c_951902a420ea.slice/crio-c82a4c86569bd3e312550ca3c8e31e4005d8bb07063f1d0c1f6ced986098fe0c WatchSource:0}: Error finding container c82a4c86569bd3e312550ca3c8e31e4005d8bb07063f1d0c1f6ced986098fe0c: Status 404 returned error can't find the container with id c82a4c86569bd3e312550ca3c8e31e4005d8bb07063f1d0c1f6ced986098fe0c Jul 08 20:06:01.284806 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:01.284772 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" event={"ID":"865977f3-fd8b-45f3-867c-951902a420ea","Type":"ContainerStarted","Data":"c82a4c86569bd3e312550ca3c8e31e4005d8bb07063f1d0c1f6ced986098fe0c"} Jul 08 20:06:03.295604 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:03.295574 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" event={"ID":"865977f3-fd8b-45f3-867c-951902a420ea","Type":"ContainerStarted","Data":"0df39b79bb3a4578cbd61b4301ddaaaeb5e0948a19d6360e16cd3335c0f8208b"} Jul 08 20:06:03.328408 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:03.328364 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-79bfv" podStartSLOduration=1.672640072 podStartE2EDuration="4.328349937s" totalImagesPullingTime="2.655709865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:05:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:06:03.327978477 +0000 UTC m=+331.324995411" watchObservedRunningTime="2026-07-08 20:06:03.328349937 +0000 UTC m=+331.325366872" Jul 08 20:06:04.148933 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:04.148902 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:04.244378 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:04.244349 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:06.223772 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:06.223727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:20.786083 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:20.786048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:20.876653 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:20.876624 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:21.218287 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.218243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:21.622021 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.621951 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:21.729050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.729004 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:21.784426 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.784408 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc"] Jul 08 20:06:21.788011 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.787995 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.792698 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.792639 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 20:06:21.792830 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.792814 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 20:06:21.792914 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.792836 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 20:06:21.792914 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.792817 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dh5bd\"" Jul 08 20:06:21.793412 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.793395 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 20:06:21.816116 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.816097 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc"] Jul 08 20:06:21.845006 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.844980 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9ecda20-8582-4251-83cb-706e1b13f347-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.845097 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.845029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9ecda20-8582-4251-83cb-706e1b13f347-webhook-cert\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.845097 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.845073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rmc\" (UniqueName: \"kubernetes.io/projected/d9ecda20-8582-4251-83cb-706e1b13f347-kube-api-access-j6rmc\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.845187 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.845099 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9ecda20-8582-4251-83cb-706e1b13f347-manifests\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.845187 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.845145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9ecda20-8582-4251-83cb-706e1b13f347-tmp\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.945720 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.945701 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9ecda20-8582-4251-83cb-706e1b13f347-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.945821 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.945746 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9ecda20-8582-4251-83cb-706e1b13f347-webhook-cert\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.945821 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.945779 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rmc\" (UniqueName: \"kubernetes.io/projected/d9ecda20-8582-4251-83cb-706e1b13f347-kube-api-access-j6rmc\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.945821 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.945810 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9ecda20-8582-4251-83cb-706e1b13f347-manifests\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.946024 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.945852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9ecda20-8582-4251-83cb-706e1b13f347-tmp\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.946252 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.946230 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d9ecda20-8582-4251-83cb-706e1b13f347-manifests\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.947871 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.947849 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9ecda20-8582-4251-83cb-706e1b13f347-tmp\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.947964 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.947948 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d9ecda20-8582-4251-83cb-706e1b13f347-webhook-cert\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.948088 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.948072 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d9ecda20-8582-4251-83cb-706e1b13f347-apiservice-cert\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:21.953522 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.953504 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:21.956642 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:21.956621 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rmc\" (UniqueName: \"kubernetes.io/projected/d9ecda20-8582-4251-83cb-706e1b13f347-kube-api-access-j6rmc\") pod \"opendatahub-operator-controller-manager-67959458dc-fdjcc\" (UID: \"d9ecda20-8582-4251-83cb-706e1b13f347\") " pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:22.098525 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.098505 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:22.102746 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.102726 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:22.223813 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.223747 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:22.249980 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.249938 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc"] Jul 08 20:06:22.253690 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:06:22.253667 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ecda20_8582_4251_83cb_706e1b13f347.slice/crio-5b5f087f04852f4ba2dc7ad43d4205e5bbe87b046cac53bc053b4a408293b0d0 WatchSource:0}: Error finding container 5b5f087f04852f4ba2dc7ad43d4205e5bbe87b046cac53bc053b4a408293b0d0: Status 404 returned error can't find the container with id 5b5f087f04852f4ba2dc7ad43d4205e5bbe87b046cac53bc053b4a408293b0d0 Jul 08 20:06:22.351468 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.351436 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:22.382046 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.382021 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" event={"ID":"d9ecda20-8582-4251-83cb-706e1b13f347","Type":"ContainerStarted","Data":"5b5f087f04852f4ba2dc7ad43d4205e5bbe87b046cac53bc053b4a408293b0d0"} Jul 08 20:06:22.449091 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.449058 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:22.555958 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.555870 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:22.712911 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.712815 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:22.813783 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:22.813706 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:23.843984 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:23.843949 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:23.965445 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:23.965410 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:25.397662 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:25.397638 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" event={"ID":"d9ecda20-8582-4251-83cb-706e1b13f347","Type":"ContainerStarted","Data":"ec0e199cd1c4f25dbf97f8e12e387531bb66dc0c8e026e072083340d6851c8d6"} Jul 08 20:06:27.408036 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:27.408006 2573 generic.go:356] "Generic (PLEG): container finished" podID="d9ecda20-8582-4251-83cb-706e1b13f347" containerID="ec0e199cd1c4f25dbf97f8e12e387531bb66dc0c8e026e072083340d6851c8d6" exitCode=0 Jul 08 20:06:27.408424 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:27.408052 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" event={"ID":"d9ecda20-8582-4251-83cb-706e1b13f347","Type":"ContainerDied","Data":"ec0e199cd1c4f25dbf97f8e12e387531bb66dc0c8e026e072083340d6851c8d6"} Jul 08 20:06:28.415747 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:28.415716 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" event={"ID":"d9ecda20-8582-4251-83cb-706e1b13f347","Type":"ContainerStarted","Data":"795bbc2978b6d6fab80864d8692e55a263a240648078b5616a86be24a0d47ae2"} Jul 08 20:06:28.416211 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:28.415961 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:28.540039 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:28.540013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:28.672531 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:28.672450 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:39.120988 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.120932 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" podStartSLOduration=13.523374851 podStartE2EDuration="18.120917112s" totalImagesPullingTime="3.539985405s" totalInitContainerRuntime="1.057556856s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:06:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:06:28.450860258 +0000 UTC m=+356.447877193" watchObservedRunningTime="2026-07-08 20:06:39.120917112 +0000 UTC m=+367.117934046" Jul 08 20:06:39.121578 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.121514 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8"] Jul 08 20:06:39.129644 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.129623 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.133719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.133697 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 20:06:39.133852 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.133699 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 20:06:39.133852 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.133701 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 20:06:39.133852 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.133701 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dnqvv\"" Jul 08 20:06:39.144092 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.144071 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8"] Jul 08 20:06:39.191424 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.191403 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ef44d1d-7d7e-4346-8c21-83d38fff475c-metrics-cert\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.191515 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.191428 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ef44d1d-7d7e-4346-8c21-83d38fff475c-cert\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.191561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.191528 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6ef44d1d-7d7e-4346-8c21-83d38fff475c-manager-config\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.191561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.191553 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84dqc\" (UniqueName: \"kubernetes.io/projected/6ef44d1d-7d7e-4346-8c21-83d38fff475c-kube-api-access-84dqc\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.291829 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.291808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ef44d1d-7d7e-4346-8c21-83d38fff475c-metrics-cert\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.291914 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.291833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ef44d1d-7d7e-4346-8c21-83d38fff475c-cert\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.291914 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.291887 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6ef44d1d-7d7e-4346-8c21-83d38fff475c-manager-config\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.291914 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.291903 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84dqc\" (UniqueName: \"kubernetes.io/projected/6ef44d1d-7d7e-4346-8c21-83d38fff475c-kube-api-access-84dqc\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.292644 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.292619 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6ef44d1d-7d7e-4346-8c21-83d38fff475c-manager-config\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.294074 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.294051 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6ef44d1d-7d7e-4346-8c21-83d38fff475c-metrics-cert\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.294150 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.294085 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ef44d1d-7d7e-4346-8c21-83d38fff475c-cert\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.300776 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.300754 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84dqc\" (UniqueName: \"kubernetes.io/projected/6ef44d1d-7d7e-4346-8c21-83d38fff475c-kube-api-access-84dqc\") pod \"lws-controller-manager-557dbd68d4-4g9g8\" (UID: \"6ef44d1d-7d7e-4346-8c21-83d38fff475c\") " pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.421663 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.421644 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-67959458dc-fdjcc" Jul 08 20:06:39.437879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.437859 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:39.567062 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:39.567036 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8"] Jul 08 20:06:39.569475 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:06:39.569444 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef44d1d_7d7e_4346_8c21_83d38fff475c.slice/crio-01dc0f35ad8e457d9c7432d1184ab58aae5867760519cd266460fad3e81eb46e WatchSource:0}: Error finding container 01dc0f35ad8e457d9c7432d1184ab58aae5867760519cd266460fad3e81eb46e: Status 404 returned error can't find the container with id 01dc0f35ad8e457d9c7432d1184ab58aae5867760519cd266460fad3e81eb46e Jul 08 20:06:40.475063 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:40.475027 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" event={"ID":"6ef44d1d-7d7e-4346-8c21-83d38fff475c","Type":"ContainerStarted","Data":"01dc0f35ad8e457d9c7432d1184ab58aae5867760519cd266460fad3e81eb46e"} Jul 08 20:06:41.481424 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:41.481390 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" event={"ID":"6ef44d1d-7d7e-4346-8c21-83d38fff475c","Type":"ContainerStarted","Data":"f4ed9717d156ed05cb29bd6b8aa1e83f9da6271bd01f125adf44e759aeab5d20"} Jul 08 20:06:41.481853 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:41.481456 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:41.492290 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:41.492270 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:41.534763 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:41.534716 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" podStartSLOduration=1.258132585 podStartE2EDuration="2.534698008s" totalImagesPullingTime="1.276565423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:06:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:06:41.531027067 +0000 UTC m=+369.528044001" watchObservedRunningTime="2026-07-08 20:06:41.534698008 +0000 UTC m=+369.531714945" Jul 08 20:06:41.646916 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:41.646893 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:42.244256 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.244229 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-789db8bf69-ftszt"] Jul 08 20:06:42.246541 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.246520 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.247295 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.247272 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:42.260522 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.260507 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 20:06:42.264641 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.264628 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 20:06:42.265758 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.265745 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 20:06:42.265850 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.265759 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-w6qqc\"" Jul 08 20:06:42.278258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.278239 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 20:06:42.304733 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.304712 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-789db8bf69-ftszt"] Jul 08 20:06:42.319135 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.319117 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597-tls-certs\") pod \"kube-auth-proxy-789db8bf69-ftszt\" (UID: \"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597\") " pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.319203 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.319168 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrhz\" (UniqueName: \"kubernetes.io/projected/df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597-kube-api-access-pqrhz\") pod \"kube-auth-proxy-789db8bf69-ftszt\" (UID: \"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597\") " pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.319251 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.319231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597-tmp\") pod \"kube-auth-proxy-789db8bf69-ftszt\" (UID: \"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597\") " pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.419722 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.419700 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597-tls-certs\") pod \"kube-auth-proxy-789db8bf69-ftszt\" (UID: \"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597\") " pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.419836 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.419819 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrhz\" (UniqueName: \"kubernetes.io/projected/df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597-kube-api-access-pqrhz\") pod \"kube-auth-proxy-789db8bf69-ftszt\" (UID: \"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597\") " pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.419879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.419861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597-tmp\") pod \"kube-auth-proxy-789db8bf69-ftszt\" (UID: \"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597\") " pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.421898 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.421881 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597-tls-certs\") pod \"kube-auth-proxy-789db8bf69-ftszt\" (UID: \"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597\") " pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.421988 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.421908 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597-tmp\") pod \"kube-auth-proxy-789db8bf69-ftszt\" (UID: \"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597\") " pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.442327 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.442297 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrhz\" (UniqueName: \"kubernetes.io/projected/df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597-kube-api-access-pqrhz\") pod \"kube-auth-proxy-789db8bf69-ftszt\" (UID: \"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597\") " pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.449224 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.449201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:42.555056 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.555005 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" Jul 08 20:06:42.700720 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.700697 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-789db8bf69-ftszt"] Jul 08 20:06:42.702322 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:06:42.702290 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5d7fb6_8f6b_4bba_bbf9_b69f7e76e597.slice/crio-7582c67764a8a49a37da3fe443ca6a44a5b6e4905a8cce13923b2eb6e8791ca7 WatchSource:0}: Error finding container 7582c67764a8a49a37da3fe443ca6a44a5b6e4905a8cce13923b2eb6e8791ca7: Status 404 returned error can't find the container with id 7582c67764a8a49a37da3fe443ca6a44a5b6e4905a8cce13923b2eb6e8791ca7 Jul 08 20:06:42.721401 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.721374 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:42.873249 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:42.873180 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:43.351443 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:43.351414 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:43.491204 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:43.491171 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" event={"ID":"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597","Type":"ContainerStarted","Data":"7582c67764a8a49a37da3fe443ca6a44a5b6e4905a8cce13923b2eb6e8791ca7"} Jul 08 20:06:43.531388 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:43.531361 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:43.712917 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:43.712859 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:44.629218 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:44.629188 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:44.748114 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:44.748090 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:46.167865 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:46.167833 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:46.507444 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:46.507412 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" event={"ID":"df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597","Type":"ContainerStarted","Data":"0ed9f5951b77fb44cb9a71e872186462b4481afda3f145e0173f34ad3a00bc57"} Jul 08 20:06:46.530683 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:46.530641 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-789db8bf69-ftszt" podStartSLOduration=0.872161481 podStartE2EDuration="4.530625944s" totalImagesPullingTime="3.658464463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:06:46.528752436 +0000 UTC m=+374.525769372" watchObservedRunningTime="2026-07-08 20:06:46.530625944 +0000 UTC m=+374.527642920" Jul 08 20:06:50.180116 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:50.180078 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:50.393905 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:50.393865 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:51.785098 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:51.785062 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:52.055033 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:52.054964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:52.258230 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:52.258204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:52.488251 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:52.488226 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-557dbd68d4-4g9g8" Jul 08 20:06:52.877469 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:52.877398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:53.059167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:53.059142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:54.987688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:54.987657 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:55.151092 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:55.151066 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:55.356723 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:55.356631 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:55.496033 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:55.496001 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:06:55.755683 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:06:55.755645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:43.210027 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:07:43.209999 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:43.326002 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:07:43.325973 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:43.526040 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:07:43.525940 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:44.215124 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:07:44.215093 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:07:44.422927 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:07:44.422896 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:13.487591 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:13.487558 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:16.875232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:16.875204 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:18.879650 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:18.879615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:20.484599 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:20.484568 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:20.872170 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:20.872099 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:21.699963 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:21.699921 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:22.669822 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.669791 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67545bd57d-fmk76"] Jul 08 20:08:22.673164 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.673146 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.695174 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.695145 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67545bd57d-fmk76"] Jul 08 20:08:22.755240 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.755210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hw2\" (UniqueName: \"kubernetes.io/projected/36af0242-cb8c-483d-a569-2bd951759e98-kube-api-access-x9hw2\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.755696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.755248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36af0242-cb8c-483d-a569-2bd951759e98-console-oauth-config\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.755696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.755338 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-oauth-serving-cert\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.755696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.755393 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36af0242-cb8c-483d-a569-2bd951759e98-console-serving-cert\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.755696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.755446 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-console-config\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.755696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.755519 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-trusted-ca-bundle\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.755696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.755551 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-service-ca\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.774145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.774110 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:22.856954 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.856930 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36af0242-cb8c-483d-a569-2bd951759e98-console-serving-cert\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.857082 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.856979 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-console-config\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.857082 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.857021 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-trusted-ca-bundle\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.857082 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.857050 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-service-ca\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.857256 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.857186 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hw2\" (UniqueName: \"kubernetes.io/projected/36af0242-cb8c-483d-a569-2bd951759e98-kube-api-access-x9hw2\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.857256 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.857230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36af0242-cb8c-483d-a569-2bd951759e98-console-oauth-config\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.857423 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.857392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-oauth-serving-cert\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.858499 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.858477 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-console-config\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.858611 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.858536 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-oauth-serving-cert\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.858673 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.858607 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-trusted-ca-bundle\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.859064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.859036 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/36af0242-cb8c-483d-a569-2bd951759e98-service-ca\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.860028 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.860007 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/36af0242-cb8c-483d-a569-2bd951759e98-console-serving-cert\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.860457 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.860440 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/36af0242-cb8c-483d-a569-2bd951759e98-console-oauth-config\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.874126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.874105 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hw2\" (UniqueName: \"kubernetes.io/projected/36af0242-cb8c-483d-a569-2bd951759e98-kube-api-access-x9hw2\") pod \"console-67545bd57d-fmk76\" (UID: \"36af0242-cb8c-483d-a569-2bd951759e98\") " pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:22.923521 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.922658 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:22.983226 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:22.983198 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:23.121560 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:23.121538 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67545bd57d-fmk76"] Jul 08 20:08:23.123326 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:08:23.123291 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36af0242_cb8c_483d_a569_2bd951759e98.slice/crio-c24821574aca4996e510d889969e47c757dddfa747cc81be3d3ceb89a4b604dc WatchSource:0}: Error finding container c24821574aca4996e510d889969e47c757dddfa747cc81be3d3ceb89a4b604dc: Status 404 returned error can't find the container with id c24821574aca4996e510d889969e47c757dddfa747cc81be3d3ceb89a4b604dc Jul 08 20:08:23.924403 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:23.924376 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:23.958183 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:23.958154 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67545bd57d-fmk76" event={"ID":"36af0242-cb8c-483d-a569-2bd951759e98","Type":"ContainerStarted","Data":"ab2c4cb7e386d3304f4342e96cab41ec9558274749094cc9f0c06330f61e03cd"} Jul 08 20:08:23.958344 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:23.958200 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67545bd57d-fmk76" event={"ID":"36af0242-cb8c-483d-a569-2bd951759e98","Type":"ContainerStarted","Data":"c24821574aca4996e510d889969e47c757dddfa747cc81be3d3ceb89a4b604dc"} Jul 08 20:08:23.994462 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:23.994417 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67545bd57d-fmk76" podStartSLOduration=1.994400078 podStartE2EDuration="1.994400078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:08:23.992801388 +0000 UTC m=+471.989818322" watchObservedRunningTime="2026-07-08 20:08:23.994400078 +0000 UTC m=+471.991417015" Jul 08 20:08:24.047417 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:24.047398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:24.150177 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:24.150148 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:25.906939 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:25.906908 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:26.470388 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.470357 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k"] Jul 08 20:08:26.473637 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.473621 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" Jul 08 20:08:26.479509 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.479490 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 20:08:26.479719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.479704 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 20:08:26.479773 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.479704 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 20:08:26.479773 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.479741 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-p6c4t\"" Jul 08 20:08:26.490405 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.490386 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k"] Jul 08 20:08:26.592903 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.592880 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc6cs\" (UniqueName: \"kubernetes.io/projected/fd398bc2-4cc4-4d1f-801f-08a7aa0322ca-kube-api-access-xc6cs\") pod \"dns-operator-controller-manager-864dd78c4c-p4x4k\" (UID: \"fd398bc2-4cc4-4d1f-801f-08a7aa0322ca\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" Jul 08 20:08:26.693634 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.693608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc6cs\" (UniqueName: \"kubernetes.io/projected/fd398bc2-4cc4-4d1f-801f-08a7aa0322ca-kube-api-access-xc6cs\") pod \"dns-operator-controller-manager-864dd78c4c-p4x4k\" (UID: \"fd398bc2-4cc4-4d1f-801f-08a7aa0322ca\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" Jul 08 20:08:26.703448 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.703424 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc6cs\" (UniqueName: \"kubernetes.io/projected/fd398bc2-4cc4-4d1f-801f-08a7aa0322ca-kube-api-access-xc6cs\") pod \"dns-operator-controller-manager-864dd78c4c-p4x4k\" (UID: \"fd398bc2-4cc4-4d1f-801f-08a7aa0322ca\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" Jul 08 20:08:26.784082 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.784023 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" Jul 08 20:08:26.916807 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.916784 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k"] Jul 08 20:08:26.918171 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:08:26.918146 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd398bc2_4cc4_4d1f_801f_08a7aa0322ca.slice/crio-e22e79abbe330a2bf188699fa8ef2d4e7b6732949f9f2870be47210203195159 WatchSource:0}: Error finding container e22e79abbe330a2bf188699fa8ef2d4e7b6732949f9f2870be47210203195159: Status 404 returned error can't find the container with id e22e79abbe330a2bf188699fa8ef2d4e7b6732949f9f2870be47210203195159 Jul 08 20:08:26.974035 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:26.974009 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" event={"ID":"fd398bc2-4cc4-4d1f-801f-08a7aa0322ca","Type":"ContainerStarted","Data":"e22e79abbe330a2bf188699fa8ef2d4e7b6732949f9f2870be47210203195159"} Jul 08 20:08:28.670546 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:28.670513 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:28.749384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:28.749352 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:29.619821 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.619789 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn"] Jul 08 20:08:29.623195 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.623177 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:29.626723 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.626694 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 08 20:08:29.626831 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.626761 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-gwkxj\"" Jul 08 20:08:29.626831 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.626777 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 08 20:08:29.629689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.629662 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:29.634730 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.634712 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn"] Jul 08 20:08:29.720695 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.720668 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:29.721048 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.720698 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4sqk\" (UniqueName: \"kubernetes.io/projected/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-kube-api-access-g4sqk\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:29.721048 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.720832 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:29.821636 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.821606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:29.821790 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.821772 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:29.821857 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.821799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4sqk\" (UniqueName: \"kubernetes.io/projected/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-kube-api-access-g4sqk\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:29.821915 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:08:29.821905 2573 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 08 20:08:29.821982 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:08:29.821970 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-plugin-serving-cert podName:7a4b6b9a-d0e7-4f22-9ded-94e70ed19098 nodeName:}" failed. No retries permitted until 2026-07-08 20:08:30.321952193 +0000 UTC m=+478.318969117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-pvzpn" (UID: "7a4b6b9a-d0e7-4f22-9ded-94e70ed19098") : secret "plugin-serving-cert" not found Jul 08 20:08:29.822360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.822337 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:29.835182 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.835159 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4sqk\" (UniqueName: \"kubernetes.io/projected/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-kube-api-access-g4sqk\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:29.882125 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:29.882068 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:30.325586 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:30.325561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:30.327843 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:30.327826 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a4b6b9a-d0e7-4f22-9ded-94e70ed19098-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-pvzpn\" (UID: \"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:30.532981 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:30.532959 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" Jul 08 20:08:30.658249 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:30.658223 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn"] Jul 08 20:08:30.659590 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:08:30.659562 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a4b6b9a_d0e7_4f22_9ded_94e70ed19098.slice/crio-cd59a53fcad50f555e967f64bc4c8c31881e9cab9d057989e6d11a721acfa939 WatchSource:0}: Error finding container cd59a53fcad50f555e967f64bc4c8c31881e9cab9d057989e6d11a721acfa939: Status 404 returned error can't find the container with id cd59a53fcad50f555e967f64bc4c8c31881e9cab9d057989e6d11a721acfa939 Jul 08 20:08:30.992893 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:30.992867 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" event={"ID":"fd398bc2-4cc4-4d1f-801f-08a7aa0322ca","Type":"ContainerStarted","Data":"65f3288254a0b1109a81ebc79fcea9c740f6bf918acf8b306bbca5eaf2d48689"} Jul 08 20:08:30.993256 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:30.992948 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" Jul 08 20:08:30.993999 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:30.993980 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" event={"ID":"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098","Type":"ContainerStarted","Data":"cd59a53fcad50f555e967f64bc4c8c31881e9cab9d057989e6d11a721acfa939"} Jul 08 20:08:31.019186 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:31.019145 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" podStartSLOduration=1.927808526 podStartE2EDuration="5.019132298s" totalImagesPullingTime="3.091323772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:08:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:08:31.01804723 +0000 UTC m=+479.015064164" watchObservedRunningTime="2026-07-08 20:08:31.019132298 +0000 UTC m=+479.016149311" Jul 08 20:08:32.983642 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:32.983611 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:32.984077 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:32.983672 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:32.988647 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:32.988622 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:33.007972 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:33.007948 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67545bd57d-fmk76" Jul 08 20:08:33.095565 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:33.095136 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597557c7d4-5fgck"] Jul 08 20:08:33.256584 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:33.256508 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:33.356801 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:33.356773 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:33.456552 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:33.456530 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:36.875440 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:36.875407 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:36.977179 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:36.977138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:40.709533 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.709499 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q"] Jul 08 20:08:40.713389 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.713371 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:08:40.717099 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.717076 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hhfqt\"" Jul 08 20:08:40.739863 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.739839 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q"] Jul 08 20:08:40.816363 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.816336 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7e46a6a-1743-48c3-8329-11ebd8788c6f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5wq5q\" (UID: \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:08:40.816469 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.816374 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtpv5\" (UniqueName: \"kubernetes.io/projected/e7e46a6a-1743-48c3-8329-11ebd8788c6f-kube-api-access-jtpv5\") pod \"kuadrant-operator-controller-manager-86878c769f-5wq5q\" (UID: \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:08:40.917584 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.917554 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtpv5\" (UniqueName: \"kubernetes.io/projected/e7e46a6a-1743-48c3-8329-11ebd8788c6f-kube-api-access-jtpv5\") pod \"kuadrant-operator-controller-manager-86878c769f-5wq5q\" (UID: \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:08:40.917748 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.917707 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7e46a6a-1743-48c3-8329-11ebd8788c6f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5wq5q\" (UID: \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:08:40.918064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.918033 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7e46a6a-1743-48c3-8329-11ebd8788c6f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-5wq5q\" (UID: \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:08:40.931798 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:40.931776 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtpv5\" (UniqueName: \"kubernetes.io/projected/e7e46a6a-1743-48c3-8329-11ebd8788c6f-kube-api-access-jtpv5\") pod \"kuadrant-operator-controller-manager-86878c769f-5wq5q\" (UID: \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:08:41.024976 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:41.024917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:08:41.178678 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:41.178657 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q"] Jul 08 20:08:41.780568 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:41.780535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:41.873549 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:41.873520 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:42.001878 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:42.001852 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4x4k" Jul 08 20:08:42.663064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:42.663032 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:44.146499 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:44.146457 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:44.249726 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:44.249687 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:50.098360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:50.098305 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:50.200515 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:50.200476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:55.089139 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:08:55.089107 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e46a6a_1743_48c3_8329_11ebd8788c6f.slice/crio-9fd141b8ff063641300d851ba374e277ec752b6d6b10a88d810302eb645ccd87 WatchSource:0}: Error finding container 9fd141b8ff063641300d851ba374e277ec752b6d6b10a88d810302eb645ccd87: Status 404 returned error can't find the container with id 9fd141b8ff063641300d851ba374e277ec752b6d6b10a88d810302eb645ccd87 Jul 08 20:08:55.133696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:55.133668 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" event={"ID":"e7e46a6a-1743-48c3-8329-11ebd8788c6f","Type":"ContainerStarted","Data":"9fd141b8ff063641300d851ba374e277ec752b6d6b10a88d810302eb645ccd87"} Jul 08 20:08:56.140285 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:56.140245 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" event={"ID":"7a4b6b9a-d0e7-4f22-9ded-94e70ed19098","Type":"ContainerStarted","Data":"5d65f970166e8345b6e91cd10b9ff41a7f76c58294563006ff5352a3ccaf2b48"} Jul 08 20:08:57.278187 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:57.278159 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:57.381173 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:57.381143 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:08:58.126674 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.126623 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-597557c7d4-5fgck" podUID="3beb52b3-cda5-48a1-a109-56b9e5afbb6f" containerName="console" containerID="cri-o://b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b" gracePeriod=15 Jul 08 20:08:58.415449 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.415425 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:08:58.445837 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.445775 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-pvzpn" podStartSLOduration=4.765921624 podStartE2EDuration="29.445754883s" totalImagesPullingTime="24.679833259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:08:56.165600236 +0000 UTC m=+504.162617175" watchObservedRunningTime="2026-07-08 20:08:58.445754883 +0000 UTC m=+506.442771818" Jul 08 20:08:58.481178 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481152 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-config\") pod \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " Jul 08 20:08:58.481305 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481200 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-oauth-config\") pod \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " Jul 08 20:08:58.481305 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481231 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-oauth-serving-cert\") pod \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " Jul 08 20:08:58.481456 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481359 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-serving-cert\") pod \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " Jul 08 20:08:58.481456 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481413 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-trusted-ca-bundle\") pod \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " Jul 08 20:08:58.481559 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481476 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57z6s\" (UniqueName: \"kubernetes.io/projected/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-kube-api-access-57z6s\") pod \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " Jul 08 20:08:58.481559 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481549 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-service-ca\") pod \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\" (UID: \"3beb52b3-cda5-48a1-a109-56b9e5afbb6f\") " Jul 08 20:08:58.481655 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481580 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-config" (OuterVolumeSpecName: "console-config") pod "3beb52b3-cda5-48a1-a109-56b9e5afbb6f" (UID: "3beb52b3-cda5-48a1-a109-56b9e5afbb6f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:08:58.481655 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481604 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3beb52b3-cda5-48a1-a109-56b9e5afbb6f" (UID: "3beb52b3-cda5-48a1-a109-56b9e5afbb6f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:08:58.482021 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481975 2573 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:08:58.482021 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.481998 2573 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-oauth-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:08:58.482021 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.482007 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3beb52b3-cda5-48a1-a109-56b9e5afbb6f" (UID: "3beb52b3-cda5-48a1-a109-56b9e5afbb6f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:08:58.482021 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.482014 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-service-ca" (OuterVolumeSpecName: "service-ca") pod "3beb52b3-cda5-48a1-a109-56b9e5afbb6f" (UID: "3beb52b3-cda5-48a1-a109-56b9e5afbb6f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:08:58.484019 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.483993 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-kube-api-access-57z6s" (OuterVolumeSpecName: "kube-api-access-57z6s") pod "3beb52b3-cda5-48a1-a109-56b9e5afbb6f" (UID: "3beb52b3-cda5-48a1-a109-56b9e5afbb6f"). InnerVolumeSpecName "kube-api-access-57z6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:08:58.484140 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.484109 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3beb52b3-cda5-48a1-a109-56b9e5afbb6f" (UID: "3beb52b3-cda5-48a1-a109-56b9e5afbb6f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:08:58.484196 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.484131 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3beb52b3-cda5-48a1-a109-56b9e5afbb6f" (UID: "3beb52b3-cda5-48a1-a109-56b9e5afbb6f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:08:58.582789 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.582753 2573 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-oauth-config\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:08:58.582789 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.582782 2573 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-console-serving-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:08:58.582972 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.582796 2573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-trusted-ca-bundle\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:08:58.582972 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.582808 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57z6s\" (UniqueName: \"kubernetes.io/projected/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-kube-api-access-57z6s\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:08:58.582972 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:58.582821 2573 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3beb52b3-cda5-48a1-a109-56b9e5afbb6f-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:08:59.158450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:59.158364 2573 generic.go:356] "Generic (PLEG): container finished" podID="3beb52b3-cda5-48a1-a109-56b9e5afbb6f" containerID="b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b" exitCode=0 Jul 08 20:08:59.158450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:59.158425 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597557c7d4-5fgck" event={"ID":"3beb52b3-cda5-48a1-a109-56b9e5afbb6f","Type":"ContainerDied","Data":"b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b"} Jul 08 20:08:59.158672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:59.158469 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597557c7d4-5fgck" event={"ID":"3beb52b3-cda5-48a1-a109-56b9e5afbb6f","Type":"ContainerDied","Data":"28980baa3bc5d8074cde4ac4ad5dc9b11e14ad28774bd48568d6e453a00964d2"} Jul 08 20:08:59.158672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:59.158499 2573 scope.go:117] "RemoveContainer" containerID="b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b" Jul 08 20:08:59.158672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:59.158425 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597557c7d4-5fgck" Jul 08 20:08:59.168386 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:59.168375 2573 scope.go:117] "RemoveContainer" containerID="b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b" Jul 08 20:08:59.168664 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:08:59.168641 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b\": container with ID starting with b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b not found: ID does not exist" containerID="b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b" Jul 08 20:08:59.168736 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:59.168673 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b"} err="failed to get container status \"b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b\": rpc error: code = NotFound desc = could not find container \"b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b\": container with ID starting with b9105d06c9ba9b2efc8ae78cd23aee025c76230bad02f55cecde7c6b15b0cf9b not found: ID does not exist" Jul 08 20:08:59.179347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:59.179321 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-597557c7d4-5fgck"] Jul 08 20:08:59.181287 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:08:59.181258 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-597557c7d4-5fgck"] Jul 08 20:09:00.786097 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:00.786053 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3beb52b3-cda5-48a1-a109-56b9e5afbb6f" path="/var/lib/kubelet/pods/3beb52b3-cda5-48a1-a109-56b9e5afbb6f/volumes" Jul 08 20:09:02.070660 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:02.070624 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:02.171048 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:02.171010 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:03.184292 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:03.184258 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" event={"ID":"e7e46a6a-1743-48c3-8329-11ebd8788c6f","Type":"ContainerStarted","Data":"ad422575fe3bba664a1ea88c9f15b962ce1f3ee3563f16376d425f8f36d9cb29"} Jul 08 20:09:03.184684 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:03.184464 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:09:03.208132 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:03.208076 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" podStartSLOduration=15.814468017 podStartE2EDuration="23.208057792s" totalImagesPullingTime="7.393589775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:09:03.206416198 +0000 UTC m=+511.203433133" watchObservedRunningTime="2026-07-08 20:09:03.208057792 +0000 UTC m=+511.205074729" Jul 08 20:09:04.522081 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:04.522048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:04.621552 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:04.621527 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:14.190891 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:14.190859 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:09:30.849232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:30.849153 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:31.062957 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:31.062931 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:33.761007 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:33.760970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:33.872341 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:33.872280 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:33.939629 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:33.939596 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:34.696462 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:34.696435 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:34.802141 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:34.802110 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:42.523038 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:42.523007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:42.622034 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:42.622007 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:50.716030 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:50.715991 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:54.215535 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:54.215490 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:54.309921 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:54.309894 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:09:54.799174 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:09:54.799147 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:08.127886 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:08.127855 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:08.408032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:08.407954 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:11.103338 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:11.103284 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:13.379789 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:13.379758 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:15.780910 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.780875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:15.813279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.813251 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8479b9b6f7-474vd"] Jul 08 20:10:15.813659 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.813646 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb52b3-cda5-48a1-a109-56b9e5afbb6f" containerName="console" Jul 08 20:10:15.813702 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.813662 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb52b3-cda5-48a1-a109-56b9e5afbb6f" containerName="console" Jul 08 20:10:15.813736 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.813732 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beb52b3-cda5-48a1-a109-56b9e5afbb6f" containerName="console" Jul 08 20:10:15.815717 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.815702 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:15.821194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.821172 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 20:10:15.821337 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.821204 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 20:10:15.821337 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.821204 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-p5v28\"" Jul 08 20:10:15.831240 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.831223 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8479b9b6f7-474vd"] Jul 08 20:10:15.891655 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.891631 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxj75\" (UniqueName: \"kubernetes.io/projected/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-kube-api-access-zxj75\") pod \"maas-controller-8479b9b6f7-474vd\" (UID: \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\") " pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:15.891777 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.891674 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-webhook-cert\") pod \"maas-controller-8479b9b6f7-474vd\" (UID: \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\") " pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:15.895790 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.895773 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:15.992783 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.992767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxj75\" (UniqueName: \"kubernetes.io/projected/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-kube-api-access-zxj75\") pod \"maas-controller-8479b9b6f7-474vd\" (UID: \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\") " pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:15.992870 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.992795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-webhook-cert\") pod \"maas-controller-8479b9b6f7-474vd\" (UID: \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\") " pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:15.994994 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:15.994978 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-webhook-cert\") pod \"maas-controller-8479b9b6f7-474vd\" (UID: \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\") " pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:16.002845 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:16.002825 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:16.004274 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:16.004258 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxj75\" (UniqueName: \"kubernetes.io/projected/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-kube-api-access-zxj75\") pod \"maas-controller-8479b9b6f7-474vd\" (UID: \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\") " pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:16.126034 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:16.125981 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:16.273044 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:16.272912 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8479b9b6f7-474vd"] Jul 08 20:10:16.275658 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:10:16.275634 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51aecd5b_83aa_4fe1_ac08_aa4bcc005c2c.slice/crio-34cb6a297bb349cbaadbbf8ca6a266b9598dabb16ca0eeef83201415e129c194 WatchSource:0}: Error finding container 34cb6a297bb349cbaadbbf8ca6a266b9598dabb16ca0eeef83201415e129c194: Status 404 returned error can't find the container with id 34cb6a297bb349cbaadbbf8ca6a266b9598dabb16ca0eeef83201415e129c194 Jul 08 20:10:16.533112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:16.533084 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8479b9b6f7-474vd" event={"ID":"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c","Type":"ContainerStarted","Data":"34cb6a297bb349cbaadbbf8ca6a266b9598dabb16ca0eeef83201415e129c194"} Jul 08 20:10:18.895919 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:18.895888 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:19.042687 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:19.042614 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:19.551495 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:19.551466 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8479b9b6f7-474vd" event={"ID":"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c","Type":"ContainerStarted","Data":"ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843"} Jul 08 20:10:19.551637 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:19.551596 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:19.576721 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:19.576678 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8479b9b6f7-474vd" podStartSLOduration=2.175704803 podStartE2EDuration="4.576665594s" totalImagesPullingTime="2.400960791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:10:19.573619465 +0000 UTC m=+587.570636407" watchObservedRunningTime="2026-07-08 20:10:19.576665594 +0000 UTC m=+587.573682534" Jul 08 20:10:29.887910 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:29.887882 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8479b9b6f7-474vd"] Jul 08 20:10:29.888427 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:29.888051 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-8479b9b6f7-474vd" podUID="51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c" containerName="manager" containerID="cri-o://ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843" gracePeriod=30 Jul 08 20:10:29.899826 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:29.899800 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:30.123685 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.123659 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:30.224243 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.224217 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxj75\" (UniqueName: \"kubernetes.io/projected/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-kube-api-access-zxj75\") pod \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\" (UID: \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\") " Jul 08 20:10:30.224401 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.224365 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-webhook-cert\") pod \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\" (UID: \"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c\") " Jul 08 20:10:30.226288 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.226262 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c" (UID: "51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:10:30.226393 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.226289 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-kube-api-access-zxj75" (OuterVolumeSpecName: "kube-api-access-zxj75") pod "51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c" (UID: "51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c"). InnerVolumeSpecName "kube-api-access-zxj75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:10:30.325473 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.325445 2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-webhook-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:10:30.325473 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.325472 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxj75\" (UniqueName: \"kubernetes.io/projected/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c-kube-api-access-zxj75\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:10:30.603521 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.603450 2573 generic.go:356] "Generic (PLEG): container finished" podID="51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c" containerID="ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843" exitCode=0 Jul 08 20:10:30.603521 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.603502 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8479b9b6f7-474vd" Jul 08 20:10:30.603672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.603533 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8479b9b6f7-474vd" event={"ID":"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c","Type":"ContainerDied","Data":"ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843"} Jul 08 20:10:30.603672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.603598 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8479b9b6f7-474vd" event={"ID":"51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c","Type":"ContainerDied","Data":"34cb6a297bb349cbaadbbf8ca6a266b9598dabb16ca0eeef83201415e129c194"} Jul 08 20:10:30.603672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.603619 2573 scope.go:117] "RemoveContainer" containerID="ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843" Jul 08 20:10:30.611995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.611977 2573 scope.go:117] "RemoveContainer" containerID="ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843" Jul 08 20:10:30.612239 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:10:30.612223 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843\": container with ID starting with ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843 not found: ID does not exist" containerID="ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843" Jul 08 20:10:30.612302 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.612246 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843"} err="failed to get container status \"ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843\": rpc error: code = NotFound desc = could not find container \"ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843\": container with ID starting with ebb9d77af8ad985088963a93b97eb1b216febfa123bff870a48611194fecb843 not found: ID does not exist" Jul 08 20:10:30.636203 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.636184 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8479b9b6f7-474vd"] Jul 08 20:10:30.639014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.638998 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8479b9b6f7-474vd"] Jul 08 20:10:30.786406 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:30.786383 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c" path="/var/lib/kubelet/pods/51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c/volumes" Jul 08 20:10:31.129404 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:31.129371 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:32.555136 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:32.555109 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:10:32.555797 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:32.555782 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:10:49.054035 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:49.053995 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:50.500036 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.500005 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-d9fpz"] Jul 08 20:10:50.500452 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.500439 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c" containerName="manager" Jul 08 20:10:50.500452 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.500453 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c" containerName="manager" Jul 08 20:10:50.500543 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.500510 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="51aecd5b-83aa-4fe1-ac08-aa4bcc005c2c" containerName="manager" Jul 08 20:10:50.503497 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.503483 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:10:50.512858 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.512838 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 08 20:10:50.518031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.518015 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-q9bx9\"" Jul 08 20:10:50.541436 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.541410 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-d9fpz"] Jul 08 20:10:50.578811 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.578787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6f9z\" (UniqueName: \"kubernetes.io/projected/fa609f0b-a36a-45ea-a12c-4ad343179ff7-kube-api-access-m6f9z\") pod \"payload-pre-processing-7964457696-d9fpz\" (UID: \"fa609f0b-a36a-45ea-a12c-4ad343179ff7\") " pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:10:50.578923 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.578872 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa609f0b-a36a-45ea-a12c-4ad343179ff7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-d9fpz\" (UID: \"fa609f0b-a36a-45ea-a12c-4ad343179ff7\") " pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:10:50.679405 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.679371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa609f0b-a36a-45ea-a12c-4ad343179ff7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-d9fpz\" (UID: \"fa609f0b-a36a-45ea-a12c-4ad343179ff7\") " pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:10:50.679511 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.679438 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6f9z\" (UniqueName: \"kubernetes.io/projected/fa609f0b-a36a-45ea-a12c-4ad343179ff7-kube-api-access-m6f9z\") pod \"payload-pre-processing-7964457696-d9fpz\" (UID: \"fa609f0b-a36a-45ea-a12c-4ad343179ff7\") " pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:10:50.680067 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.680049 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa609f0b-a36a-45ea-a12c-4ad343179ff7-plugins-config-volume\") pod \"payload-pre-processing-7964457696-d9fpz\" (UID: \"fa609f0b-a36a-45ea-a12c-4ad343179ff7\") " pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:10:50.691649 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.691625 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6f9z\" (UniqueName: \"kubernetes.io/projected/fa609f0b-a36a-45ea-a12c-4ad343179ff7-kube-api-access-m6f9z\") pod \"payload-pre-processing-7964457696-d9fpz\" (UID: \"fa609f0b-a36a-45ea-a12c-4ad343179ff7\") " pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:10:50.756575 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.756510 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:10:50.813432 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.813410 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:10:50.952094 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.952069 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-d9fpz"] Jul 08 20:10:50.953251 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:10:50.953225 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa609f0b_a36a_45ea_a12c_4ad343179ff7.slice/crio-5a2a9d6007d69fc589300978fb03879666bd96cf136c1d9399efeb97508636f9 WatchSource:0}: Error finding container 5a2a9d6007d69fc589300978fb03879666bd96cf136c1d9399efeb97508636f9: Status 404 returned error can't find the container with id 5a2a9d6007d69fc589300978fb03879666bd96cf136c1d9399efeb97508636f9 Jul 08 20:10:50.955009 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:50.954992 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:10:51.715187 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:51.715159 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" event={"ID":"fa609f0b-a36a-45ea-a12c-4ad343179ff7","Type":"ContainerStarted","Data":"5a2a9d6007d69fc589300978fb03879666bd96cf136c1d9399efeb97508636f9"} Jul 08 20:10:52.720880 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:52.720846 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" event={"ID":"fa609f0b-a36a-45ea-a12c-4ad343179ff7","Type":"ContainerStarted","Data":"00848246bc6d3eae77875ac2352ff075d1c146a41f531376e8fdf07bb6887ef7"} Jul 08 20:10:52.721293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:52.720899 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:10:52.741994 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:10:52.741950 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" podStartSLOduration=1.292750429 podStartE2EDuration="2.741936027s" totalImagesPullingTime="1.449185598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:10:52.739580855 +0000 UTC m=+620.736597791" watchObservedRunningTime="2026-07-08 20:10:52.741936027 +0000 UTC m=+620.738952983" Jul 08 20:11:01.099372 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:01.099264 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:11:01.193505 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:01.193470 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:11:03.728173 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:03.728139 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-d9fpz" Jul 08 20:11:13.404559 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.404524 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n"] Jul 08 20:11:13.407868 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.407847 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.411833 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.411804 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l89df\"" Jul 08 20:11:13.411833 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.411823 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 20:11:13.411833 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.411816 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 08 20:11:13.412054 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.412041 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 20:11:13.426642 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.426620 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n"] Jul 08 20:11:13.461273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.461252 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:11:13.561519 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.561497 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.561631 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.561561 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abbf65ad-09da-489a-91b3-b8d241fdcc7b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.561679 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.561629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.561679 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.561665 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.561759 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.561686 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5h54\" (UniqueName: \"kubernetes.io/projected/abbf65ad-09da-489a-91b3-b8d241fdcc7b-kube-api-access-f5h54\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.561835 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.561764 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.663182 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.663103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.663305 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.663183 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abbf65ad-09da-489a-91b3-b8d241fdcc7b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.663305 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.663233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.663305 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.663272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.663305 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.663293 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5h54\" (UniqueName: \"kubernetes.io/projected/abbf65ad-09da-489a-91b3-b8d241fdcc7b-kube-api-access-f5h54\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.663559 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.663360 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.663931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.663680 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.663931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.663737 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.663931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.663885 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.665647 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.665613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abbf65ad-09da-489a-91b3-b8d241fdcc7b-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.666599 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.666578 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abbf65ad-09da-489a-91b3-b8d241fdcc7b-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.674160 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.674136 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5h54\" (UniqueName: \"kubernetes.io/projected/abbf65ad-09da-489a-91b3-b8d241fdcc7b-kube-api-access-f5h54\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n\" (UID: \"abbf65ad-09da-489a-91b3-b8d241fdcc7b\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.717974 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.717954 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:13.855141 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:13.855118 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n"] Jul 08 20:11:13.857137 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:11:13.857114 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbf65ad_09da_489a_91b3_b8d241fdcc7b.slice/crio-e920a3393ca1f8d7be8dc3197ea8dded5af039a0d9b06ca5666842e8f7115757 WatchSource:0}: Error finding container e920a3393ca1f8d7be8dc3197ea8dded5af039a0d9b06ca5666842e8f7115757: Status 404 returned error can't find the container with id e920a3393ca1f8d7be8dc3197ea8dded5af039a0d9b06ca5666842e8f7115757 Jul 08 20:11:14.833792 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:14.833756 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" event={"ID":"abbf65ad-09da-489a-91b3-b8d241fdcc7b","Type":"ContainerStarted","Data":"e920a3393ca1f8d7be8dc3197ea8dded5af039a0d9b06ca5666842e8f7115757"} Jul 08 20:11:15.841255 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:15.841225 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" event={"ID":"abbf65ad-09da-489a-91b3-b8d241fdcc7b","Type":"ContainerStarted","Data":"12516de003a0eee58b13f5f1f438092a5a046b77ab93e6aff6453764988f42ba"} Jul 08 20:11:15.859862 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:15.859818 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" podStartSLOduration=1.6741340930000002 podStartE2EDuration="2.85980346s" totalImagesPullingTime="1.185669367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:11:15.858899177 +0000 UTC m=+643.855916113" watchObservedRunningTime="2026-07-08 20:11:15.85980346 +0000 UTC m=+643.856820429" Jul 08 20:11:23.609436 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:23.609394 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:11:23.719014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:23.718981 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:23.719170 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:23.719059 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:23.731021 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:23.730988 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:23.891688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:23.891630 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n" Jul 08 20:11:30.086018 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:30.085983 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:11:40.205800 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:40.205764 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:11:47.344910 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:11:47.344879 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:12:18.194877 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.194849 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt"] Jul 08 20:12:18.198245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.198229 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.201027 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.201009 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 08 20:12:18.208001 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.207977 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt"] Jul 08 20:12:18.219281 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.219259 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:12:18.265949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.265922 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.266053 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.265992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d69e6794-faa6-4a77-a58a-903281206b6a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.266053 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.266033 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.266168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.266059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hcmw\" (UniqueName: \"kubernetes.io/projected/d69e6794-faa6-4a77-a58a-903281206b6a-kube-api-access-6hcmw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.266168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.266096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.266168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.266126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.367270 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.367246 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.367398 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.367302 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d69e6794-faa6-4a77-a58a-903281206b6a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.367398 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.367372 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.367515 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.367399 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcmw\" (UniqueName: \"kubernetes.io/projected/d69e6794-faa6-4a77-a58a-903281206b6a-kube-api-access-6hcmw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.367515 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.367437 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.367515 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.367479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.367672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.367639 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.367729 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.367711 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.367783 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.367767 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.369493 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.369475 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d69e6794-faa6-4a77-a58a-903281206b6a-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.369685 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.369667 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d69e6794-faa6-4a77-a58a-903281206b6a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.375407 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.375383 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcmw\" (UniqueName: \"kubernetes.io/projected/d69e6794-faa6-4a77-a58a-903281206b6a-kube-api-access-6hcmw\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-brpzt\" (UID: \"d69e6794-faa6-4a77-a58a-903281206b6a\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.508984 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.508923 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:18.638675 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:18.638640 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt"] Jul 08 20:12:18.640820 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:12:18.640790 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69e6794_faa6_4a77_a58a_903281206b6a.slice/crio-64208b1bfde93607a08ba17b5efcb36ab6d017de7c5f05d928358a6de9df956c WatchSource:0}: Error finding container 64208b1bfde93607a08ba17b5efcb36ab6d017de7c5f05d928358a6de9df956c: Status 404 returned error can't find the container with id 64208b1bfde93607a08ba17b5efcb36ab6d017de7c5f05d928358a6de9df956c Jul 08 20:12:19.151344 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:19.151243 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" event={"ID":"d69e6794-faa6-4a77-a58a-903281206b6a","Type":"ContainerStarted","Data":"602f3c1ae529e022f32a0963eafb1ee5df89d8222157a99167509c0ff0ff717e"} Jul 08 20:12:19.151553 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:19.151485 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" event={"ID":"d69e6794-faa6-4a77-a58a-903281206b6a","Type":"ContainerStarted","Data":"64208b1bfde93607a08ba17b5efcb36ab6d017de7c5f05d928358a6de9df956c"} Jul 08 20:12:19.183531 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:19.183483 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" podStartSLOduration=0.978760752 podStartE2EDuration="1.183469922s" totalImagesPullingTime="204.70917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:12:19.181390606 +0000 UTC m=+707.178407540" watchObservedRunningTime="2026-07-08 20:12:19.183469922 +0000 UTC m=+707.180486857" Jul 08 20:12:28.509354 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:28.509259 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:28.509354 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:28.509333 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:28.521216 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:28.521192 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:28.958665 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:28.958638 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj"] Jul 08 20:12:28.962100 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:28.962081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:28.964950 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:28.964932 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 08 20:12:28.974293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:28.974271 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj"] Jul 08 20:12:29.003725 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.003703 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:12:29.058826 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.058802 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bxf\" (UniqueName: \"kubernetes.io/projected/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-kube-api-access-n2bxf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.058928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.058836 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.058928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.058870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.059045 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.058940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.059045 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.058992 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.059138 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.059074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.159726 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.159703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.159811 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.159744 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bxf\" (UniqueName: \"kubernetes.io/projected/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-kube-api-access-n2bxf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.159811 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.159774 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.159883 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.159820 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.159883 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.159853 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.159946 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.159902 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.160130 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.160108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.160180 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.160146 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.160218 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.160209 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.162029 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.162003 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.162111 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.162096 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.167365 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.167345 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bxf\" (UniqueName: \"kubernetes.io/projected/8b852c64-d879-4a4a-9eb7-7535b98ab5bf-kube-api-access-n2bxf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj\" (UID: \"8b852c64-d879-4a4a-9eb7-7535b98ab5bf\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.208864 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.208801 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-brpzt" Jul 08 20:12:29.271902 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.271881 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:29.406232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:29.405981 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj"] Jul 08 20:12:30.204713 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:30.204669 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" event={"ID":"8b852c64-d879-4a4a-9eb7-7535b98ab5bf","Type":"ContainerStarted","Data":"36083195f12b47ffa275eca795296f288383122b9e78522a97a93d305db35c44"} Jul 08 20:12:30.205085 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:30.204717 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" event={"ID":"8b852c64-d879-4a4a-9eb7-7535b98ab5bf","Type":"ContainerStarted","Data":"87dce5b4c97e05a09435d15efb2c3c9550cdba25b92a165bb0eb1e387981f817"} Jul 08 20:12:30.223794 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:30.223752 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" podStartSLOduration=1.978552184 podStartE2EDuration="2.223738099s" totalImagesPullingTime="245.185915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:12:30.221637545 +0000 UTC m=+718.218654478" watchObservedRunningTime="2026-07-08 20:12:30.223738099 +0000 UTC m=+718.220755034" Jul 08 20:12:39.272028 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:39.271998 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:39.272400 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:39.272042 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:39.283861 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:39.283838 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:12:40.265865 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:12:40.265835 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj" Jul 08 20:13:13.268817 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:13:13.268784 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:13:13.326853 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:13:13.326832 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:13:35.334029 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:13:35.333998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:13:35.433761 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:13:35.433737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:13:36.553112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:13:36.553079 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:13:36.655997 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:13:36.655970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:13:49.503612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:13:49.503579 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:14:49.154033 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:14:49.153997 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:14:50.855153 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:14:50.855117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:31.924321 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:31.924235 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:32.073296 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:32.073260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:32.173732 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:32.173701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:32.588614 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:32.588589 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:15:32.588768 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:32.588596 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:15:37.762290 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:37.762257 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:37.814719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:37.814691 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:37.975480 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:37.975455 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:38.096367 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:38.096280 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:38.194356 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:38.194299 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:43.987184 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:43.987152 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:15:44.042528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:15:44.042500 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:16:08.852760 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:16:08.852731 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:17:01.233008 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:17:01.232915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:17:50.906453 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:17:50.906425 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:18:10.229640 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:18:10.229598 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:18:34.546492 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:18:34.546413 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:19:35.753120 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:35.753086 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q"] Jul 08 20:19:35.753658 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:35.753270 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" podUID="e7e46a6a-1743-48c3-8329-11ebd8788c6f" containerName="manager" containerID="cri-o://ad422575fe3bba664a1ea88c9f15b962ce1f3ee3563f16376d425f8f36d9cb29" gracePeriod=10 Jul 08 20:19:38.315342 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:38.315292 2573 generic.go:356] "Generic (PLEG): container finished" podID="e7e46a6a-1743-48c3-8329-11ebd8788c6f" containerID="ad422575fe3bba664a1ea88c9f15b962ce1f3ee3563f16376d425f8f36d9cb29" exitCode=0 Jul 08 20:19:38.315809 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:38.315362 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" event={"ID":"e7e46a6a-1743-48c3-8329-11ebd8788c6f","Type":"ContainerDied","Data":"ad422575fe3bba664a1ea88c9f15b962ce1f3ee3563f16376d425f8f36d9cb29"} Jul 08 20:19:38.401942 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:38.401919 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:19:38.466007 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:38.465978 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7e46a6a-1743-48c3-8329-11ebd8788c6f-extensions-socket-volume\") pod \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\" (UID: \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\") " Jul 08 20:19:38.466164 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:38.466019 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtpv5\" (UniqueName: \"kubernetes.io/projected/e7e46a6a-1743-48c3-8329-11ebd8788c6f-kube-api-access-jtpv5\") pod \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\" (UID: \"e7e46a6a-1743-48c3-8329-11ebd8788c6f\") " Jul 08 20:19:38.466433 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:38.466408 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e46a6a-1743-48c3-8329-11ebd8788c6f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e7e46a6a-1743-48c3-8329-11ebd8788c6f" (UID: "e7e46a6a-1743-48c3-8329-11ebd8788c6f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:19:38.468111 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:38.468091 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e46a6a-1743-48c3-8329-11ebd8788c6f-kube-api-access-jtpv5" (OuterVolumeSpecName: "kube-api-access-jtpv5") pod "e7e46a6a-1743-48c3-8329-11ebd8788c6f" (UID: "e7e46a6a-1743-48c3-8329-11ebd8788c6f"). InnerVolumeSpecName "kube-api-access-jtpv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:19:38.567551 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:38.567476 2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7e46a6a-1743-48c3-8329-11ebd8788c6f-extensions-socket-volume\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:19:38.567551 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:38.567503 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtpv5\" (UniqueName: \"kubernetes.io/projected/e7e46a6a-1743-48c3-8329-11ebd8788c6f-kube-api-access-jtpv5\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:19:39.321698 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:39.321666 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" event={"ID":"e7e46a6a-1743-48c3-8329-11ebd8788c6f","Type":"ContainerDied","Data":"9fd141b8ff063641300d851ba374e277ec752b6d6b10a88d810302eb645ccd87"} Jul 08 20:19:39.322101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:39.321726 2573 scope.go:117] "RemoveContainer" containerID="ad422575fe3bba664a1ea88c9f15b962ce1f3ee3563f16376d425f8f36d9cb29" Jul 08 20:19:39.322101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:39.321679 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q" Jul 08 20:19:39.340678 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:39.340657 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q"] Jul 08 20:19:39.344020 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:39.343999 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-5wq5q"] Jul 08 20:19:40.787361 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:19:40.787331 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e46a6a-1743-48c3-8329-11ebd8788c6f" path="/var/lib/kubelet/pods/e7e46a6a-1743-48c3-8329-11ebd8788c6f/volumes" Jul 08 20:20:32.617867 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:20:32.617837 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:20:32.619892 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:20:32.619870 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:23:03.482245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:23:03.482206 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:23:47.876561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:23:47.876515 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:25:32.647177 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:25:32.647152 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:25:32.651345 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:25:32.651303 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:26:03.715889 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:03.715862 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:26:03.802635 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:03.802613 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:26:31.209949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:31.209915 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:26:38.830455 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:38.830425 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:26:45.258903 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:45.258872 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:26:47.127542 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:47.127507 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:26:52.337935 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:52.337904 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:26:55.518871 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:55.518833 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:26:55.614689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:55.614655 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:26:56.028882 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:26:56.028852 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:01.711104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.711073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:01.769888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.769859 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:01.898717 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.898681 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p"] Jul 08 20:27:01.899123 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.899111 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e46a6a-1743-48c3-8329-11ebd8788c6f" containerName="manager" Jul 08 20:27:01.899181 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.899126 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e46a6a-1743-48c3-8329-11ebd8788c6f" containerName="manager" Jul 08 20:27:01.899221 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.899185 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e46a6a-1743-48c3-8329-11ebd8788c6f" containerName="manager" Jul 08 20:27:01.902510 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.902490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:01.905155 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.905136 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 20:27:01.905236 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.905178 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-72dc81d2-openshift-default-dockercfg-kzgfp\"" Jul 08 20:27:01.932299 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.932276 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p"] Jul 08 20:27:01.943432 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:01.943410 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:02.003068 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.003003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-credential-socket\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.003068 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.003037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-envoy\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.003068 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.003059 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-token\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.003284 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.003169 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-podinfo\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.003284 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.003210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-certs\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.003284 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.003267 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc7zz\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-kube-api-access-bc7zz\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.003429 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.003304 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee80a1-9abc-4329-a780-6e13e574d37b-istiod-ca-cert\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.003429 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.003346 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-socket\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.003429 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.003370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-data\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.074497 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.074476 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:02.104260 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.104242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-token\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.104394 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.104285 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-podinfo\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.104394 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.104328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-certs\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.104394 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.104355 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc7zz\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-kube-api-access-bc7zz\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.104394 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.104380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee80a1-9abc-4329-a780-6e13e574d37b-istiod-ca-cert\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.104394 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.104395 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-socket\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.104634 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.104417 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-data\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.104634 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.104472 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-credential-socket\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.104634 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.104489 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-envoy\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.105051 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.105017 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-socket\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.105150 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.105066 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-data\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.105150 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.105137 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-certs\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.105224 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.105178 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-credential-socket\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.105349 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.105332 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee80a1-9abc-4329-a780-6e13e574d37b-istiod-ca-cert\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.106689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.106663 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-envoy\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.106766 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.106716 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-podinfo\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.112593 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.112569 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-token\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.112776 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.112757 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc7zz\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-kube-api-access-bc7zz\") pod \"e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.212728 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.212707 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:02.345186 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.345160 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p"] Jul 08 20:27:02.347267 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:27:02.347244 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38ee80a1_9abc_4329_a780_6e13e574d37b.slice/crio-aa6fca8a58d797675767e941741bc4839c530d56ff869cf35ca6b9bbca2539b9 WatchSource:0}: Error finding container aa6fca8a58d797675767e941741bc4839c530d56ff869cf35ca6b9bbca2539b9: Status 404 returned error can't find the container with id aa6fca8a58d797675767e941741bc4839c530d56ff869cf35ca6b9bbca2539b9 Jul 08 20:27:02.349227 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.349210 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:27:02.396229 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:02.396205 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" event={"ID":"38ee80a1-9abc-4329-a780-6e13e574d37b","Type":"ContainerStarted","Data":"aa6fca8a58d797675767e941741bc4839c530d56ff869cf35ca6b9bbca2539b9"} Jul 08 20:27:03.088450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.088396 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:03.169803 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.169770 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv"] Jul 08 20:27:03.173885 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.173856 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.175448 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.175410 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:03.176657 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.176626 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x22jf\"" Jul 08 20:27:03.176813 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.176801 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:27:03.176987 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.176957 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-72dc81d2-serving-cert\"" Jul 08 20:27:03.177661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.177637 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:27:03.181165 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.181143 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv"] Jul 08 20:27:03.214744 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.214711 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjggj\" (UniqueName: \"kubernetes.io/projected/cee118fc-db0b-48e8-94a8-2952305dc374-kube-api-access-hjggj\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.214871 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.214777 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee118fc-db0b-48e8-94a8-2952305dc374-maas-api-tls\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.214916 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.214903 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee118fc-db0b-48e8-94a8-2952305dc374-openshift-service-ca\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.248974 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.248902 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:03.316452 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.316420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee118fc-db0b-48e8-94a8-2952305dc374-openshift-service-ca\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.316607 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.316517 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjggj\" (UniqueName: \"kubernetes.io/projected/cee118fc-db0b-48e8-94a8-2952305dc374-kube-api-access-hjggj\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.316607 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.316564 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee118fc-db0b-48e8-94a8-2952305dc374-maas-api-tls\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.316723 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:27:03.316702 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-serving-cert: secret "maas-api-e2e-ait-72dc81d2-serving-cert" not found Jul 08 20:27:03.316778 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:27:03.316769 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cee118fc-db0b-48e8-94a8-2952305dc374-maas-api-tls podName:cee118fc-db0b-48e8-94a8-2952305dc374 nodeName:}" failed. No retries permitted until 2026-07-08 20:27:03.81674856 +0000 UTC m=+1591.813765480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cee118fc-db0b-48e8-94a8-2952305dc374-maas-api-tls") pod "maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" (UID: "cee118fc-db0b-48e8-94a8-2952305dc374") : secret "maas-api-e2e-ait-72dc81d2-serving-cert" not found Jul 08 20:27:03.317151 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.317126 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee118fc-db0b-48e8-94a8-2952305dc374-openshift-service-ca\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.326544 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.326503 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjggj\" (UniqueName: \"kubernetes.io/projected/cee118fc-db0b-48e8-94a8-2952305dc374-kube-api-access-hjggj\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.601117 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.601084 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:03.789398 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.789282 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:03.822674 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.822645 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee118fc-db0b-48e8-94a8-2952305dc374-maas-api-tls\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.825288 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.825263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee118fc-db0b-48e8-94a8-2952305dc374-maas-api-tls\") pod \"maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:03.888197 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:03.888134 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:04.091035 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:04.091000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:04.889443 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:04.889409 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:27:04.889542 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:04.889480 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:27:04.889542 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:04.889508 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:27:04.947727 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:04.947706 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv"] Jul 08 20:27:04.950051 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:27:04.950026 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee118fc_db0b_48e8_94a8_2952305dc374.slice/crio-3b54a6e6097346ef0b73e847e449463d1e6c30a320131a4988b923b4600ceded WatchSource:0}: Error finding container 3b54a6e6097346ef0b73e847e449463d1e6c30a320131a4988b923b4600ceded: Status 404 returned error can't find the container with id 3b54a6e6097346ef0b73e847e449463d1e6c30a320131a4988b923b4600ceded Jul 08 20:27:05.415723 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:05.415686 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" event={"ID":"cee118fc-db0b-48e8-94a8-2952305dc374","Type":"ContainerStarted","Data":"3b54a6e6097346ef0b73e847e449463d1e6c30a320131a4988b923b4600ceded"} Jul 08 20:27:05.417101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:05.417077 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" event={"ID":"38ee80a1-9abc-4329-a780-6e13e574d37b","Type":"ContainerStarted","Data":"16985491d668908f1362202afc16c3ae7404d2db178c493bb3b74e6be9e6ffca"} Jul 08 20:27:05.440074 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:05.440022 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" podStartSLOduration=1.9002569710000001 podStartE2EDuration="4.440001746s" totalImagesPullingTime="2.539744775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:27:05.436830536 +0000 UTC m=+1593.433847471" watchObservedRunningTime="2026-07-08 20:27:05.440001746 +0000 UTC m=+1593.437018682" Jul 08 20:27:05.769384 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:05.769355 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:06.213093 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:06.213063 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:06.218725 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:06.218703 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:06.423256 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:06.423230 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:06.424084 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:06.424063 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:07.428684 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:07.428656 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" event={"ID":"cee118fc-db0b-48e8-94a8-2952305dc374","Type":"ContainerStarted","Data":"68e57341e65bb019dc5f514ff0593d4e96168decf22e7ea3471232f70a90bc26"} Jul 08 20:27:07.429051 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:07.428822 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:07.445258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:07.445211 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" podStartSLOduration=2.075385339 podStartE2EDuration="4.44519487s" totalImagesPullingTime="2.369809531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:27:07.4443953 +0000 UTC m=+1595.441412234" watchObservedRunningTime="2026-07-08 20:27:07.44519487 +0000 UTC m=+1595.442211805" Jul 08 20:27:08.341515 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:08.341453 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:08.434722 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:08.434694 2573 generic.go:356] "Generic (PLEG): container finished" podID="cee118fc-db0b-48e8-94a8-2952305dc374" containerID="68e57341e65bb019dc5f514ff0593d4e96168decf22e7ea3471232f70a90bc26" exitCode=1 Jul 08 20:27:08.435155 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:08.434775 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" event={"ID":"cee118fc-db0b-48e8-94a8-2952305dc374","Type":"ContainerDied","Data":"68e57341e65bb019dc5f514ff0593d4e96168decf22e7ea3471232f70a90bc26"} Jul 08 20:27:08.435155 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:08.435008 2573 scope.go:117] "RemoveContainer" containerID="68e57341e65bb019dc5f514ff0593d4e96168decf22e7ea3471232f70a90bc26" Jul 08 20:27:08.525383 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:08.525356 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p"] Jul 08 20:27:08.547196 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:08.547167 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:08.686150 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:08.686114 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv"] Jul 08 20:27:08.698266 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:08.698243 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:08.801751 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:08.801724 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:09.454952 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.447252 2573 generic.go:356] "Generic (PLEG): container finished" podID="cee118fc-db0b-48e8-94a8-2952305dc374" containerID="4a85e739fe101838803cb607692006dd42f5085e2cb34b7dd989fb58a7eebd1c" exitCode=1 Jul 08 20:27:09.454952 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.447609 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" podUID="38ee80a1-9abc-4329-a780-6e13e574d37b" containerName="istio-proxy" containerID="cri-o://16985491d668908f1362202afc16c3ae7404d2db178c493bb3b74e6be9e6ffca" gracePeriod=30 Jul 08 20:27:09.454952 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.448183 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" event={"ID":"cee118fc-db0b-48e8-94a8-2952305dc374","Type":"ContainerDied","Data":"4a85e739fe101838803cb607692006dd42f5085e2cb34b7dd989fb58a7eebd1c"} Jul 08 20:27:09.454952 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.448282 2573 scope.go:117] "RemoveContainer" containerID="68e57341e65bb019dc5f514ff0593d4e96168decf22e7ea3471232f70a90bc26" Jul 08 20:27:09.619916 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.619894 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:09.680796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.680766 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee118fc-db0b-48e8-94a8-2952305dc374-openshift-service-ca\") pod \"cee118fc-db0b-48e8-94a8-2952305dc374\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " Jul 08 20:27:09.680951 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.680824 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee118fc-db0b-48e8-94a8-2952305dc374-maas-api-tls\") pod \"cee118fc-db0b-48e8-94a8-2952305dc374\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " Jul 08 20:27:09.680951 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.680847 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjggj\" (UniqueName: \"kubernetes.io/projected/cee118fc-db0b-48e8-94a8-2952305dc374-kube-api-access-hjggj\") pod \"cee118fc-db0b-48e8-94a8-2952305dc374\" (UID: \"cee118fc-db0b-48e8-94a8-2952305dc374\") " Jul 08 20:27:09.681166 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.681139 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee118fc-db0b-48e8-94a8-2952305dc374-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cee118fc-db0b-48e8-94a8-2952305dc374" (UID: "cee118fc-db0b-48e8-94a8-2952305dc374"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:27:09.683016 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.682988 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cee118fc-db0b-48e8-94a8-2952305dc374-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cee118fc-db0b-48e8-94a8-2952305dc374" (UID: "cee118fc-db0b-48e8-94a8-2952305dc374"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:27:09.683104 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.683036 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee118fc-db0b-48e8-94a8-2952305dc374-kube-api-access-hjggj" (OuterVolumeSpecName: "kube-api-access-hjggj") pod "cee118fc-db0b-48e8-94a8-2952305dc374" (UID: "cee118fc-db0b-48e8-94a8-2952305dc374"). InnerVolumeSpecName "kube-api-access-hjggj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:27:09.781888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.781811 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cee118fc-db0b-48e8-94a8-2952305dc374-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:09.781888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.781841 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cee118fc-db0b-48e8-94a8-2952305dc374-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:09.781888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:09.781849 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjggj\" (UniqueName: \"kubernetes.io/projected/cee118fc-db0b-48e8-94a8-2952305dc374-kube-api-access-hjggj\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:10.127908 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:10.127882 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:10.456847 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:10.456808 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" event={"ID":"cee118fc-db0b-48e8-94a8-2952305dc374","Type":"ContainerDied","Data":"3b54a6e6097346ef0b73e847e449463d1e6c30a320131a4988b923b4600ceded"} Jul 08 20:27:10.457348 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:10.456891 2573 scope.go:117] "RemoveContainer" containerID="4a85e739fe101838803cb607692006dd42f5085e2cb34b7dd989fb58a7eebd1c" Jul 08 20:27:10.457348 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:10.457038 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv" Jul 08 20:27:10.484589 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:10.484567 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv"] Jul 08 20:27:10.488595 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:10.488576 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72dc81d2-96bf58dfc-85kvv"] Jul 08 20:27:10.786801 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:10.786731 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cee118fc-db0b-48e8-94a8-2952305dc374" path="/var/lib/kubelet/pods/cee118fc-db0b-48e8-94a8-2952305dc374/volumes" Jul 08 20:27:14.348654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.348615 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:14.450021 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.449985 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:14.483286 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.483257 2573 generic.go:356] "Generic (PLEG): container finished" podID="38ee80a1-9abc-4329-a780-6e13e574d37b" containerID="16985491d668908f1362202afc16c3ae7404d2db178c493bb3b74e6be9e6ffca" exitCode=0 Jul 08 20:27:14.483546 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.483345 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" event={"ID":"38ee80a1-9abc-4329-a780-6e13e574d37b","Type":"ContainerDied","Data":"16985491d668908f1362202afc16c3ae7404d2db178c493bb3b74e6be9e6ffca"} Jul 08 20:27:14.567502 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.567475 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:14.606712 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.606662 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:14.633660 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.633625 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:14.728580 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.728548 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-podinfo\") pod \"38ee80a1-9abc-4329-a780-6e13e574d37b\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " Jul 08 20:27:14.728748 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.728608 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-envoy\") pod \"38ee80a1-9abc-4329-a780-6e13e574d37b\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " Jul 08 20:27:14.728748 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.728668 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-token\") pod \"38ee80a1-9abc-4329-a780-6e13e574d37b\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " Jul 08 20:27:14.728748 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.728692 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee80a1-9abc-4329-a780-6e13e574d37b-istiod-ca-cert\") pod \"38ee80a1-9abc-4329-a780-6e13e574d37b\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " Jul 08 20:27:14.728748 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.728721 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-socket\") pod \"38ee80a1-9abc-4329-a780-6e13e574d37b\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " Jul 08 20:27:14.728976 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.728771 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bc7zz\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-kube-api-access-bc7zz\") pod \"38ee80a1-9abc-4329-a780-6e13e574d37b\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " Jul 08 20:27:14.728976 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.728799 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-credential-socket\") pod \"38ee80a1-9abc-4329-a780-6e13e574d37b\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " Jul 08 20:27:14.728976 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.728828 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-data\") pod \"38ee80a1-9abc-4329-a780-6e13e574d37b\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " Jul 08 20:27:14.728976 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.728859 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-certs\") pod \"38ee80a1-9abc-4329-a780-6e13e574d37b\" (UID: \"38ee80a1-9abc-4329-a780-6e13e574d37b\") " Jul 08 20:27:14.729190 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.729076 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38ee80a1-9abc-4329-a780-6e13e574d37b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38ee80a1-9abc-4329-a780-6e13e574d37b" (UID: "38ee80a1-9abc-4329-a780-6e13e574d37b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:27:14.729247 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.729218 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38ee80a1-9abc-4329-a780-6e13e574d37b-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:14.729374 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.729339 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38ee80a1-9abc-4329-a780-6e13e574d37b" (UID: "38ee80a1-9abc-4329-a780-6e13e574d37b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:14.729455 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.729435 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38ee80a1-9abc-4329-a780-6e13e574d37b" (UID: "38ee80a1-9abc-4329-a780-6e13e574d37b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:14.729513 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.729450 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38ee80a1-9abc-4329-a780-6e13e574d37b" (UID: "38ee80a1-9abc-4329-a780-6e13e574d37b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:14.729657 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.729624 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-data" (OuterVolumeSpecName: "istio-data") pod "38ee80a1-9abc-4329-a780-6e13e574d37b" (UID: "38ee80a1-9abc-4329-a780-6e13e574d37b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:14.730893 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.730868 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38ee80a1-9abc-4329-a780-6e13e574d37b" (UID: "38ee80a1-9abc-4329-a780-6e13e574d37b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:14.730978 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.730895 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-token" (OuterVolumeSpecName: "istio-token") pod "38ee80a1-9abc-4329-a780-6e13e574d37b" (UID: "38ee80a1-9abc-4329-a780-6e13e574d37b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:27:14.731089 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.731075 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-kube-api-access-bc7zz" (OuterVolumeSpecName: "kube-api-access-bc7zz") pod "38ee80a1-9abc-4329-a780-6e13e574d37b" (UID: "38ee80a1-9abc-4329-a780-6e13e574d37b"). InnerVolumeSpecName "kube-api-access-bc7zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:27:14.731165 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.731147 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38ee80a1-9abc-4329-a780-6e13e574d37b" (UID: "38ee80a1-9abc-4329-a780-6e13e574d37b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:27:14.830418 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.830383 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:14.830418 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.830410 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:14.830418 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.830423 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:14.830707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.830434 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:14.830707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.830446 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:14.830707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.830458 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:14.830707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.830469 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38ee80a1-9abc-4329-a780-6e13e574d37b-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:14.830707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:14.830480 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc7zz\" (UniqueName: \"kubernetes.io/projected/38ee80a1-9abc-4329-a780-6e13e574d37b-kube-api-access-bc7zz\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:15.489466 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.489436 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" event={"ID":"38ee80a1-9abc-4329-a780-6e13e574d37b","Type":"ContainerDied","Data":"aa6fca8a58d797675767e941741bc4839c530d56ff869cf35ca6b9bbca2539b9"} Jul 08 20:27:15.489888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.489471 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p" Jul 08 20:27:15.489888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.489528 2573 scope.go:117] "RemoveContainer" containerID="16985491d668908f1362202afc16c3ae7404d2db178c493bb3b74e6be9e6ffca" Jul 08 20:27:15.507937 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.507903 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p"] Jul 08 20:27:15.511542 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.511519 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-72dc81d2-openshift-default-5899879cc-mfg4p"] Jul 08 20:27:15.685777 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.685748 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:15.768938 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.768861 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:15.774972 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.774947 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4"] Jul 08 20:27:15.775358 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.775345 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee118fc-db0b-48e8-94a8-2952305dc374" containerName="maas-api" Jul 08 20:27:15.775358 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.775359 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee118fc-db0b-48e8-94a8-2952305dc374" containerName="maas-api" Jul 08 20:27:15.775446 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.775371 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38ee80a1-9abc-4329-a780-6e13e574d37b" containerName="istio-proxy" Jul 08 20:27:15.775446 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.775378 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="38ee80a1-9abc-4329-a780-6e13e574d37b" containerName="istio-proxy" Jul 08 20:27:15.775446 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.775401 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cee118fc-db0b-48e8-94a8-2952305dc374" containerName="maas-api" Jul 08 20:27:15.775446 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.775407 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cee118fc-db0b-48e8-94a8-2952305dc374" containerName="maas-api" Jul 08 20:27:15.775565 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.775475 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee118fc-db0b-48e8-94a8-2952305dc374" containerName="maas-api" Jul 08 20:27:15.775565 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.775487 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="38ee80a1-9abc-4329-a780-6e13e574d37b" containerName="istio-proxy" Jul 08 20:27:15.775565 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.775496 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cee118fc-db0b-48e8-94a8-2952305dc374" containerName="maas-api" Jul 08 20:27:15.778171 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.778155 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:15.781638 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.781613 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x22jf\"" Jul 08 20:27:15.781786 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.781640 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bb4eb1cc-serving-cert\"" Jul 08 20:27:15.781786 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.781640 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:27:15.782137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.782125 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:27:15.793155 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.793136 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4"] Jul 08 20:27:15.834503 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.834472 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:15.841713 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.841685 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlrb\" (UniqueName: \"kubernetes.io/projected/4eccb745-782a-4f9d-b0a9-ae383778da6d-kube-api-access-5vlrb\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:15.841823 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.841773 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eccb745-782a-4f9d-b0a9-ae383778da6d-openshift-service-ca\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:15.841931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.841913 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eccb745-782a-4f9d-b0a9-ae383778da6d-maas-api-tls\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:15.942924 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.942888 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eccb745-782a-4f9d-b0a9-ae383778da6d-openshift-service-ca\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:15.943125 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.942943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eccb745-782a-4f9d-b0a9-ae383778da6d-maas-api-tls\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:15.943125 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.942996 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlrb\" (UniqueName: \"kubernetes.io/projected/4eccb745-782a-4f9d-b0a9-ae383778da6d-kube-api-access-5vlrb\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:15.943125 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:27:15.943100 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-serving-cert: secret "maas-api-e2e-ait-bb4eb1cc-serving-cert" not found Jul 08 20:27:15.943293 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:27:15.943167 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eccb745-782a-4f9d-b0a9-ae383778da6d-maas-api-tls podName:4eccb745-782a-4f9d-b0a9-ae383778da6d nodeName:}" failed. No retries permitted until 2026-07-08 20:27:16.443146834 +0000 UTC m=+1604.440163747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4eccb745-782a-4f9d-b0a9-ae383778da6d-maas-api-tls") pod "maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" (UID: "4eccb745-782a-4f9d-b0a9-ae383778da6d") : secret "maas-api-e2e-ait-bb4eb1cc-serving-cert" not found Jul 08 20:27:15.943643 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.943620 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eccb745-782a-4f9d-b0a9-ae383778da6d-openshift-service-ca\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:15.953841 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:15.953822 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlrb\" (UniqueName: \"kubernetes.io/projected/4eccb745-782a-4f9d-b0a9-ae383778da6d-kube-api-access-5vlrb\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:16.037966 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:16.037882 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:16.348766 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:16.348679 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:16.448051 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:16.448012 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eccb745-782a-4f9d-b0a9-ae383778da6d-maas-api-tls\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:16.450445 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:16.450426 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eccb745-782a-4f9d-b0a9-ae383778da6d-maas-api-tls\") pod \"maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:16.451666 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:16.451641 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:16.689638 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:16.689609 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:16.788784 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:16.788745 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38ee80a1-9abc-4329-a780-6e13e574d37b" path="/var/lib/kubelet/pods/38ee80a1-9abc-4329-a780-6e13e574d37b/volumes" Jul 08 20:27:16.820121 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:16.820098 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4"] Jul 08 20:27:16.822704 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:27:16.822678 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eccb745_782a_4f9d_b0a9_ae383778da6d.slice/crio-92a54af9fff57d43956d3b24ab922a3d02224e50a9ddced248f3522669897d23 WatchSource:0}: Error finding container 92a54af9fff57d43956d3b24ab922a3d02224e50a9ddced248f3522669897d23: Status 404 returned error can't find the container with id 92a54af9fff57d43956d3b24ab922a3d02224e50a9ddced248f3522669897d23 Jul 08 20:27:17.500707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:17.500616 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" event={"ID":"4eccb745-782a-4f9d-b0a9-ae383778da6d","Type":"ContainerStarted","Data":"c04f3800b85ade7072c3ec12dab1c6510e99665fb277188c84c12904f99ff557"} Jul 08 20:27:17.500707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:17.500667 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" event={"ID":"4eccb745-782a-4f9d-b0a9-ae383778da6d","Type":"ContainerStarted","Data":"92a54af9fff57d43956d3b24ab922a3d02224e50a9ddced248f3522669897d23"} Jul 08 20:27:17.501098 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:17.500742 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:17.518240 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:17.518196 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" podStartSLOduration=2.132560261 podStartE2EDuration="2.518177268s" totalImagesPullingTime="385.617007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:27:17.517013565 +0000 UTC m=+1605.514030500" watchObservedRunningTime="2026-07-08 20:27:17.518177268 +0000 UTC m=+1605.515194203" Jul 08 20:27:18.283458 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:18.283421 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:18.506850 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:18.506817 2573 generic.go:356] "Generic (PLEG): container finished" podID="4eccb745-782a-4f9d-b0a9-ae383778da6d" containerID="c04f3800b85ade7072c3ec12dab1c6510e99665fb277188c84c12904f99ff557" exitCode=1 Jul 08 20:27:18.507017 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:18.506905 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" event={"ID":"4eccb745-782a-4f9d-b0a9-ae383778da6d","Type":"ContainerDied","Data":"c04f3800b85ade7072c3ec12dab1c6510e99665fb277188c84c12904f99ff557"} Jul 08 20:27:18.507141 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:18.507128 2573 scope.go:117] "RemoveContainer" containerID="c04f3800b85ade7072c3ec12dab1c6510e99665fb277188c84c12904f99ff557" Jul 08 20:27:19.513619 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:19.513588 2573 generic.go:356] "Generic (PLEG): container finished" podID="4eccb745-782a-4f9d-b0a9-ae383778da6d" containerID="151e49b9a5edfcc67b5529d3f82b76a0aab2d553b6cddf6c375fb0a59ddc605a" exitCode=1 Jul 08 20:27:19.514037 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:19.513682 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" event={"ID":"4eccb745-782a-4f9d-b0a9-ae383778da6d","Type":"ContainerDied","Data":"151e49b9a5edfcc67b5529d3f82b76a0aab2d553b6cddf6c375fb0a59ddc605a"} Jul 08 20:27:19.514037 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:19.513756 2573 scope.go:117] "RemoveContainer" containerID="c04f3800b85ade7072c3ec12dab1c6510e99665fb277188c84c12904f99ff557" Jul 08 20:27:19.514037 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:19.513883 2573 scope.go:117] "RemoveContainer" containerID="151e49b9a5edfcc67b5529d3f82b76a0aab2d553b6cddf6c375fb0a59ddc605a" Jul 08 20:27:19.514156 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:27:19.514098 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4_odh-ai-gateway-infra(4eccb745-782a-4f9d-b0a9-ae383778da6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" podUID="4eccb745-782a-4f9d-b0a9-ae383778da6d" Jul 08 20:27:20.521234 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:20.521206 2573 scope.go:117] "RemoveContainer" containerID="151e49b9a5edfcc67b5529d3f82b76a0aab2d553b6cddf6c375fb0a59ddc605a" Jul 08 20:27:20.521618 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:27:20.521458 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4_odh-ai-gateway-infra(4eccb745-782a-4f9d-b0a9-ae383778da6d)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" podUID="4eccb745-782a-4f9d-b0a9-ae383778da6d" Jul 08 20:27:20.929588 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:20.929557 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:21.295844 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.295760 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4"] Jul 08 20:27:21.325267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.325237 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:21.657143 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.657123 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:21.797228 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.797197 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eccb745-782a-4f9d-b0a9-ae383778da6d-maas-api-tls\") pod \"4eccb745-782a-4f9d-b0a9-ae383778da6d\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " Jul 08 20:27:21.797414 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.797298 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eccb745-782a-4f9d-b0a9-ae383778da6d-openshift-service-ca\") pod \"4eccb745-782a-4f9d-b0a9-ae383778da6d\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " Jul 08 20:27:21.797414 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.797357 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vlrb\" (UniqueName: \"kubernetes.io/projected/4eccb745-782a-4f9d-b0a9-ae383778da6d-kube-api-access-5vlrb\") pod \"4eccb745-782a-4f9d-b0a9-ae383778da6d\" (UID: \"4eccb745-782a-4f9d-b0a9-ae383778da6d\") " Jul 08 20:27:21.797649 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.797624 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eccb745-782a-4f9d-b0a9-ae383778da6d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4eccb745-782a-4f9d-b0a9-ae383778da6d" (UID: "4eccb745-782a-4f9d-b0a9-ae383778da6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:27:21.799229 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.799207 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eccb745-782a-4f9d-b0a9-ae383778da6d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4eccb745-782a-4f9d-b0a9-ae383778da6d" (UID: "4eccb745-782a-4f9d-b0a9-ae383778da6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:27:21.799351 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.799239 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eccb745-782a-4f9d-b0a9-ae383778da6d-kube-api-access-5vlrb" (OuterVolumeSpecName: "kube-api-access-5vlrb") pod "4eccb745-782a-4f9d-b0a9-ae383778da6d" (UID: "4eccb745-782a-4f9d-b0a9-ae383778da6d"). InnerVolumeSpecName "kube-api-access-5vlrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:27:21.898719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.898632 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eccb745-782a-4f9d-b0a9-ae383778da6d-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:21.898719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.898665 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vlrb\" (UniqueName: \"kubernetes.io/projected/4eccb745-782a-4f9d-b0a9-ae383778da6d-kube-api-access-5vlrb\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:21.898719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:21.898681 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4eccb745-782a-4f9d-b0a9-ae383778da6d-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:22.532415 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:22.532387 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" Jul 08 20:27:22.532606 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:22.532387 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4" event={"ID":"4eccb745-782a-4f9d-b0a9-ae383778da6d","Type":"ContainerDied","Data":"92a54af9fff57d43956d3b24ab922a3d02224e50a9ddced248f3522669897d23"} Jul 08 20:27:22.532606 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:22.532515 2573 scope.go:117] "RemoveContainer" containerID="151e49b9a5edfcc67b5529d3f82b76a0aab2d553b6cddf6c375fb0a59ddc605a" Jul 08 20:27:22.553393 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:22.553366 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4"] Jul 08 20:27:22.555043 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:22.555025 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bb4eb1cc-5f58bdc786-hpnt4"] Jul 08 20:27:22.787753 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:22.787677 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eccb745-782a-4f9d-b0a9-ae383778da6d" path="/var/lib/kubelet/pods/4eccb745-782a-4f9d-b0a9-ae383778da6d/volumes" Jul 08 20:27:27.143983 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:27.143907 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:27.284177 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:27.284144 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:33.087141 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.087059 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:33.151764 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.151738 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:33.282801 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.282777 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx"] Jul 08 20:27:33.283275 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.283255 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eccb745-782a-4f9d-b0a9-ae383778da6d" containerName="maas-api" Jul 08 20:27:33.283364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.283282 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eccb745-782a-4f9d-b0a9-ae383778da6d" containerName="maas-api" Jul 08 20:27:33.283433 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.283421 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eccb745-782a-4f9d-b0a9-ae383778da6d" containerName="maas-api" Jul 08 20:27:33.283472 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.283438 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eccb745-782a-4f9d-b0a9-ae383778da6d" containerName="maas-api" Jul 08 20:27:33.286498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.286484 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.289258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.289129 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 20:27:33.289258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.289147 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-1d6dccd6-openshift-default-dockercfg-2hqc2\"" Jul 08 20:27:33.298784 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.298759 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx"] Jul 08 20:27:33.325603 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.325575 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:33.407872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.407812 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbzh4\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-kube-api-access-nbzh4\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.407872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.407848 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-credential-socket\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.407992 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.407872 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-token\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.408032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.407997 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-data\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.408065 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.408029 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-envoy\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.408065 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.408048 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-podinfo\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.408137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.408066 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istiod-ca-cert\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.408137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.408110 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-certs\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.408207 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.408154 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-socket\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509270 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-certs\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509394 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-socket\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509394 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509369 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbzh4\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-kube-api-access-nbzh4\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509474 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-credential-socket\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509474 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509420 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-token\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509551 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509516 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-data\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509598 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509560 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-envoy\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509598 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-podinfo\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509702 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509614 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istiod-ca-cert\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509702 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509665 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-certs\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509798 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509706 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-socket\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509855 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509813 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-credential-socket\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.509855 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.509818 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-data\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.510144 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.510125 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istiod-ca-cert\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.511614 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.511584 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-envoy\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.511816 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.511799 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-podinfo\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.520214 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.520194 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbzh4\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-kube-api-access-nbzh4\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.520480 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.520460 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-token\") pod \"e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.599287 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.599266 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:33.733406 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.733382 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx"] Jul 08 20:27:33.735072 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:27:33.735050 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434bab19_fbc9_43d4_9f75_4017c2cea7eb.slice/crio-c439cb2201107bb567d10c8a54f4005292e9c1291dbd3caa100c62fcd11a8364 WatchSource:0}: Error finding container c439cb2201107bb567d10c8a54f4005292e9c1291dbd3caa100c62fcd11a8364: Status 404 returned error can't find the container with id c439cb2201107bb567d10c8a54f4005292e9c1291dbd3caa100c62fcd11a8364 Jul 08 20:27:33.737168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.737137 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:27:33.737247 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.737199 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:27:33.737247 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:33.737225 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:27:34.419769 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.419739 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:34.502494 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.502471 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:34.568581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.568556 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:34.591144 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.591123 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" event={"ID":"434bab19-fbc9-43d4-9f75-4017c2cea7eb","Type":"ContainerStarted","Data":"97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb"} Jul 08 20:27:34.591248 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.591169 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" event={"ID":"434bab19-fbc9-43d4-9f75-4017c2cea7eb","Type":"ContainerStarted","Data":"c439cb2201107bb567d10c8a54f4005292e9c1291dbd3caa100c62fcd11a8364"} Jul 08 20:27:34.600034 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.599937 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:34.614210 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.614173 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" podStartSLOduration=1.614159651 podStartE2EDuration="1.614159651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:27:34.612739549 +0000 UTC m=+1622.609756496" watchObservedRunningTime="2026-07-08 20:27:34.614159651 +0000 UTC m=+1622.611176586" Jul 08 20:27:34.618774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.618754 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:34.729480 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.729426 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:34.959226 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:34.959192 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:35.075219 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:35.075147 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:35.173857 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:35.173835 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:35.274219 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:35.274197 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:35.596289 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:35.596269 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:35.597218 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:35.597202 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:38.286568 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:38.286542 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:38.939453 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:38.939418 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:39.101961 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:39.101920 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx"] Jul 08 20:27:39.102159 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:39.102137 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" podUID="434bab19-fbc9-43d4-9f75-4017c2cea7eb" containerName="istio-proxy" containerID="cri-o://97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb" gracePeriod=30 Jul 08 20:27:39.129061 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:39.129031 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:39.228889 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:39.228813 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:39.333211 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:39.333181 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:41.017802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:41.017773 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:44.344822 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.344801 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:44.401998 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.401942 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-socket\") pod \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " Jul 08 20:27:44.401998 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.401973 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-podinfo\") pod \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " Jul 08 20:27:44.401998 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.401992 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-credential-socket\") pod \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " Jul 08 20:27:44.402192 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402053 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-certs\") pod \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " Jul 08 20:27:44.402192 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-envoy\") pod \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " Jul 08 20:27:44.402192 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402106 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbzh4\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-kube-api-access-nbzh4\") pod \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " Jul 08 20:27:44.402192 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402134 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istiod-ca-cert\") pod \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " Jul 08 20:27:44.402192 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402179 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-data\") pod \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " Jul 08 20:27:44.402530 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402211 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-token\") pod \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\" (UID: \"434bab19-fbc9-43d4-9f75-4017c2cea7eb\") " Jul 08 20:27:44.402530 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402209 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "434bab19-fbc9-43d4-9f75-4017c2cea7eb" (UID: "434bab19-fbc9-43d4-9f75-4017c2cea7eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:44.402530 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402367 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "434bab19-fbc9-43d4-9f75-4017c2cea7eb" (UID: "434bab19-fbc9-43d4-9f75-4017c2cea7eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:44.402530 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402383 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "434bab19-fbc9-43d4-9f75-4017c2cea7eb" (UID: "434bab19-fbc9-43d4-9f75-4017c2cea7eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:44.402719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402564 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "434bab19-fbc9-43d4-9f75-4017c2cea7eb" (UID: "434bab19-fbc9-43d4-9f75-4017c2cea7eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:44.402719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402571 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:44.402719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402596 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:44.402719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402610 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:44.402719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.402655 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "434bab19-fbc9-43d4-9f75-4017c2cea7eb" (UID: "434bab19-fbc9-43d4-9f75-4017c2cea7eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:27:44.404225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.404199 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "434bab19-fbc9-43d4-9f75-4017c2cea7eb" (UID: "434bab19-fbc9-43d4-9f75-4017c2cea7eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:27:44.404348 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.404252 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-kube-api-access-nbzh4" (OuterVolumeSpecName: "kube-api-access-nbzh4") pod "434bab19-fbc9-43d4-9f75-4017c2cea7eb" (UID: "434bab19-fbc9-43d4-9f75-4017c2cea7eb"). InnerVolumeSpecName "kube-api-access-nbzh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:27:44.404513 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.404496 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "434bab19-fbc9-43d4-9f75-4017c2cea7eb" (UID: "434bab19-fbc9-43d4-9f75-4017c2cea7eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:27:44.404570 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.404527 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "434bab19-fbc9-43d4-9f75-4017c2cea7eb" (UID: "434bab19-fbc9-43d4-9f75-4017c2cea7eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:27:44.503327 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.503294 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:44.503414 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.503329 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:44.503414 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.503342 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:44.503414 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.503353 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:44.503414 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.503365 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbzh4\" (UniqueName: \"kubernetes.io/projected/434bab19-fbc9-43d4-9f75-4017c2cea7eb-kube-api-access-nbzh4\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:44.503414 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.503376 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/434bab19-fbc9-43d4-9f75-4017c2cea7eb-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:27:44.639572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.639543 2573 generic.go:356] "Generic (PLEG): container finished" podID="434bab19-fbc9-43d4-9f75-4017c2cea7eb" containerID="97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb" exitCode=0 Jul 08 20:27:44.639654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.639580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" event={"ID":"434bab19-fbc9-43d4-9f75-4017c2cea7eb","Type":"ContainerDied","Data":"97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb"} Jul 08 20:27:44.639654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.639596 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" Jul 08 20:27:44.639654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.639630 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx" event={"ID":"434bab19-fbc9-43d4-9f75-4017c2cea7eb","Type":"ContainerDied","Data":"c439cb2201107bb567d10c8a54f4005292e9c1291dbd3caa100c62fcd11a8364"} Jul 08 20:27:44.639754 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.639660 2573 scope.go:117] "RemoveContainer" containerID="97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb" Jul 08 20:27:44.648046 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.648026 2573 scope.go:117] "RemoveContainer" containerID="97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb" Jul 08 20:27:44.648290 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:27:44.648268 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb\": container with ID starting with 97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb not found: ID does not exist" containerID="97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb" Jul 08 20:27:44.648395 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.648299 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb"} err="failed to get container status \"97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb\": rpc error: code = NotFound desc = could not find container \"97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb\": container with ID starting with 97f82e6d6599d50380b3d1c39203e5ec3e1ef25da98355ec7c4c05dd528bb7eb not found: ID does not exist" Jul 08 20:27:44.660256 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.660207 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx"] Jul 08 20:27:44.666176 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.666159 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-1d6dccd6-openshift-default-64c65cbf95-k5mwx"] Jul 08 20:27:44.786266 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:44.786247 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="434bab19-fbc9-43d4-9f75-4017c2cea7eb" path="/var/lib/kubelet/pods/434bab19-fbc9-43d4-9f75-4017c2cea7eb/volumes" Jul 08 20:27:45.130602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:45.130576 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:45.230753 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:45.230733 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:45.659785 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:45.659761 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:45.757139 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:45.757114 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:45.889481 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:45.889456 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:47.410065 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:47.410033 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:52.981192 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:52.981158 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:53.091267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:53.091238 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:53.185141 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:53.185114 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:53.280814 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:53.280760 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:27:53.751137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:27:53.751099 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:20.008995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:20.008964 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:25.701479 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:25.701449 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:25.802932 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:25.802904 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:25.901772 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:25.901747 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:41.887228 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:41.887195 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:47.609665 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:47.609630 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:47.714500 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:47.714451 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:47.810443 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:47.810412 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:58.408382 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:58.408293 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:28:59.274515 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:28:59.274483 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:04.994870 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:04.994831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:05.053120 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:05.053090 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:05.134735 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:05.134708 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:05.233687 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:05.233666 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:05.335395 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:05.335334 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:06.161443 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:06.161408 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:06.259782 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:06.259759 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:06.367494 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:06.367468 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:08.149714 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:08.149676 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:13.762805 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:13.762771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:13.823451 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:13.823417 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:14.495602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:14.495567 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:20.351535 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.351498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:20.452606 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.452571 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:20.611509 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.611419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:20.924355 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.924322 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq"] Jul 08 20:29:20.925186 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.925164 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="434bab19-fbc9-43d4-9f75-4017c2cea7eb" containerName="istio-proxy" Jul 08 20:29:20.925186 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.925186 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="434bab19-fbc9-43d4-9f75-4017c2cea7eb" containerName="istio-proxy" Jul 08 20:29:20.925389 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.925216 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eccb745-782a-4f9d-b0a9-ae383778da6d" containerName="maas-api" Jul 08 20:29:20.925389 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.925225 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eccb745-782a-4f9d-b0a9-ae383778da6d" containerName="maas-api" Jul 08 20:29:20.925480 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.925421 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="434bab19-fbc9-43d4-9f75-4017c2cea7eb" containerName="istio-proxy" Jul 08 20:29:20.929301 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.929276 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:20.931926 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.931895 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 20:29:20.932066 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.932049 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-76d496a5-openshift-default-dockercfg-sthx9\"" Jul 08 20:29:20.945305 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.945280 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq"] Jul 08 20:29:20.952650 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:20.952612 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:21.004250 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.004228 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-certs\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.004362 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.004258 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-istio-token\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.004362 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.004287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-data\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.004362 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.004304 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-envoy\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.004478 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.004389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ba3330-29e6-439e-af48-739177845682-istiod-ca-cert\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.004478 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.004435 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ba3330-29e6-439e-af48-739177845682-istio-podinfo\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.004478 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.004452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-socket\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.004478 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.004473 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86qth\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-kube-api-access-86qth\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.004611 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.004510 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-credential-socket\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.096397 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.096374 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:21.105867 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.105849 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ba3330-29e6-439e-af48-739177845682-istiod-ca-cert\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.105949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.105886 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ba3330-29e6-439e-af48-739177845682-istio-podinfo\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.105949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.105904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-socket\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.105949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.105932 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qth\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-kube-api-access-86qth\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106072 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.105957 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-credential-socket\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106072 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.106024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-certs\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106072 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.106048 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-istio-token\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.106077 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-data\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106225 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.106101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-envoy\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106439 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.106419 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-certs\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106514 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.106489 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-data\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106600 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.106573 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-socket\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.106610 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-credential-socket\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.106707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.106624 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ba3330-29e6-439e-af48-739177845682-istiod-ca-cert\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.108368 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.108352 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-envoy\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.108436 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.108400 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ba3330-29e6-439e-af48-739177845682-istio-podinfo\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.113424 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.113406 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86qth\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-kube-api-access-86qth\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.113495 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.113435 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-istio-token\") pod \"e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.239247 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.239194 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:21.373374 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.373348 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq"] Jul 08 20:29:21.375396 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:29:21.375373 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ba3330_29e6_439e_af48_739177845682.slice/crio-6ea28dcb1031fbb44d66304cd667afe835fef3999ff457965ba108e9aa4695e1 WatchSource:0}: Error finding container 6ea28dcb1031fbb44d66304cd667afe835fef3999ff457965ba108e9aa4695e1: Status 404 returned error can't find the container with id 6ea28dcb1031fbb44d66304cd667afe835fef3999ff457965ba108e9aa4695e1 Jul 08 20:29:21.377667 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.377638 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:29:21.377759 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.377715 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:29:21.377828 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:21.377759 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:29:22.106630 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:22.106589 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" event={"ID":"20ba3330-29e6-439e-af48-739177845682","Type":"ContainerStarted","Data":"c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd"} Jul 08 20:29:22.106810 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:22.106646 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" event={"ID":"20ba3330-29e6-439e-af48-739177845682","Type":"ContainerStarted","Data":"6ea28dcb1031fbb44d66304cd667afe835fef3999ff457965ba108e9aa4695e1"} Jul 08 20:29:22.125463 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:22.125408 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" podStartSLOduration=2.125392809 podStartE2EDuration="2.125392809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:29:22.124810624 +0000 UTC m=+1730.121827572" watchObservedRunningTime="2026-07-08 20:29:22.125392809 +0000 UTC m=+1730.122409743" Jul 08 20:29:22.239709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:22.239679 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:22.245042 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:22.245016 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:23.111168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:23.111139 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:23.112233 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:23.112215 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:24.124876 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.124827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:24.199432 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.199405 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6"] Jul 08 20:29:24.203060 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.203036 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.205547 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.205523 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:24.205730 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.205637 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-76d496a5-serving-cert\"" Jul 08 20:29:24.205878 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.205764 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:29:24.206729 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.206712 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:29:24.206835 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.206719 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x22jf\"" Jul 08 20:29:24.211730 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.211709 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6"] Jul 08 20:29:24.278014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.277987 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:24.338347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.338300 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fca6b2-34d0-450f-8317-2d175b5b293f-openshift-service-ca\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.338488 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.338356 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fca6b2-34d0-450f-8317-2d175b5b293f-maas-api-tls\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.338488 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.338476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47zs\" (UniqueName: \"kubernetes.io/projected/25fca6b2-34d0-450f-8317-2d175b5b293f-kube-api-access-k47zs\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.439936 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.439915 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fca6b2-34d0-450f-8317-2d175b5b293f-openshift-service-ca\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.440054 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.439950 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fca6b2-34d0-450f-8317-2d175b5b293f-maas-api-tls\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.440054 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.440024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47zs\" (UniqueName: \"kubernetes.io/projected/25fca6b2-34d0-450f-8317-2d175b5b293f-kube-api-access-k47zs\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.440141 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:29:24.440119 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-serving-cert: secret "maas-api-e2e-mt-76d496a5-serving-cert" not found Jul 08 20:29:24.440194 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:29:24.440184 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25fca6b2-34d0-450f-8317-2d175b5b293f-maas-api-tls podName:25fca6b2-34d0-450f-8317-2d175b5b293f nodeName:}" failed. No retries permitted until 2026-07-08 20:29:24.940167356 +0000 UTC m=+1732.937184270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/25fca6b2-34d0-450f-8317-2d175b5b293f-maas-api-tls") pod "maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" (UID: "25fca6b2-34d0-450f-8317-2d175b5b293f") : secret "maas-api-e2e-mt-76d496a5-serving-cert" not found Jul 08 20:29:24.440577 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.440561 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fca6b2-34d0-450f-8317-2d175b5b293f-openshift-service-ca\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.448387 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.448368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47zs\" (UniqueName: \"kubernetes.io/projected/25fca6b2-34d0-450f-8317-2d175b5b293f-kube-api-access-k47zs\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.515799 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.515770 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:24.695421 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.695331 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:24.798705 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.798675 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:24.946242 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.946160 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fca6b2-34d0-450f-8317-2d175b5b293f-maas-api-tls\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:24.948222 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:24.948204 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fca6b2-34d0-450f-8317-2d175b5b293f-maas-api-tls\") pod \"maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:25.116737 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:25.116707 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:25.250239 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:25.250213 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6"] Jul 08 20:29:25.252378 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:29:25.252350 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fca6b2_34d0_450f_8317_2d175b5b293f.slice/crio-15213d6aa178e47e7d3f0002918c25f6a1cd2cac1ed22432fff7f64102b41e84 WatchSource:0}: Error finding container 15213d6aa178e47e7d3f0002918c25f6a1cd2cac1ed22432fff7f64102b41e84: Status 404 returned error can't find the container with id 15213d6aa178e47e7d3f0002918c25f6a1cd2cac1ed22432fff7f64102b41e84 Jul 08 20:29:26.133214 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:26.133125 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" event={"ID":"25fca6b2-34d0-450f-8317-2d175b5b293f","Type":"ContainerStarted","Data":"042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3"} Jul 08 20:29:26.133214 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:26.133192 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:26.133214 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:26.133209 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" event={"ID":"25fca6b2-34d0-450f-8317-2d175b5b293f","Type":"ContainerStarted","Data":"15213d6aa178e47e7d3f0002918c25f6a1cd2cac1ed22432fff7f64102b41e84"} Jul 08 20:29:26.151654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:26.151600 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" podStartSLOduration=1.655112272 podStartE2EDuration="2.151581922s" totalImagesPullingTime="496.46965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:29:26.150770554 +0000 UTC m=+1734.147787489" watchObservedRunningTime="2026-07-08 20:29:26.151581922 +0000 UTC m=+1734.148598859" Jul 08 20:29:28.689904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:28.689876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:29.429333 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:29.429268 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:29.527424 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:29.527391 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:29.911084 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:29.911008 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:32.146491 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:32.146409 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:35.190172 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:35.190140 2573 generic.go:356] "Generic (PLEG): container finished" podID="20ba3330-29e6-439e-af48-739177845682" containerID="c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd" exitCode=0 Jul 08 20:29:35.190597 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:35.190213 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" event={"ID":"20ba3330-29e6-439e-af48-739177845682","Type":"ContainerDied","Data":"c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd"} Jul 08 20:29:35.190639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:35.190618 2573 scope.go:117] "RemoveContainer" containerID="c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd" Jul 08 20:29:35.191689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:35.191664 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:29:35.191778 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:35.191761 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:29:35.191843 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:35.191830 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:29:35.764262 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:35.764234 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:35.866688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:35.866649 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:36.194379 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.194344 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:36.197084 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.197052 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" event={"ID":"20ba3330-29e6-439e-af48-739177845682","Type":"ContainerStarted","Data":"f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717"} Jul 08 20:29:36.239556 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.239525 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:36.240608 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.240580 2573 patch_prober.go:28] interesting pod/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 08 20:29:36.240722 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.240621 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 08 20:29:36.537290 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.537199 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6"] Jul 08 20:29:36.537479 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.537454 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" podUID="25fca6b2-34d0-450f-8317-2d175b5b293f" containerName="maas-api" containerID="cri-o://042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3" gracePeriod=30 Jul 08 20:29:36.556730 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.556698 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:36.656351 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.656298 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:36.822126 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.822099 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:36.871335 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.870831 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fca6b2-34d0-450f-8317-2d175b5b293f-openshift-service-ca\") pod \"25fca6b2-34d0-450f-8317-2d175b5b293f\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " Jul 08 20:29:36.871335 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.870921 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k47zs\" (UniqueName: \"kubernetes.io/projected/25fca6b2-34d0-450f-8317-2d175b5b293f-kube-api-access-k47zs\") pod \"25fca6b2-34d0-450f-8317-2d175b5b293f\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " Jul 08 20:29:36.871335 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.870953 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fca6b2-34d0-450f-8317-2d175b5b293f-maas-api-tls\") pod \"25fca6b2-34d0-450f-8317-2d175b5b293f\" (UID: \"25fca6b2-34d0-450f-8317-2d175b5b293f\") " Jul 08 20:29:36.872099 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.872045 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fca6b2-34d0-450f-8317-2d175b5b293f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "25fca6b2-34d0-450f-8317-2d175b5b293f" (UID: "25fca6b2-34d0-450f-8317-2d175b5b293f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:29:36.873646 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.873616 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fca6b2-34d0-450f-8317-2d175b5b293f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "25fca6b2-34d0-450f-8317-2d175b5b293f" (UID: "25fca6b2-34d0-450f-8317-2d175b5b293f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:29:36.874033 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.874013 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fca6b2-34d0-450f-8317-2d175b5b293f-kube-api-access-k47zs" (OuterVolumeSpecName: "kube-api-access-k47zs") pod "25fca6b2-34d0-450f-8317-2d175b5b293f" (UID: "25fca6b2-34d0-450f-8317-2d175b5b293f"). InnerVolumeSpecName "kube-api-access-k47zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:29:36.972061 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.972022 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25fca6b2-34d0-450f-8317-2d175b5b293f-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:36.972061 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.972058 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k47zs\" (UniqueName: \"kubernetes.io/projected/25fca6b2-34d0-450f-8317-2d175b5b293f-kube-api-access-k47zs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:36.972061 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:36.972067 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25fca6b2-34d0-450f-8317-2d175b5b293f-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:37.203618 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.203588 2573 generic.go:356] "Generic (PLEG): container finished" podID="25fca6b2-34d0-450f-8317-2d175b5b293f" containerID="042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3" exitCode=0 Jul 08 20:29:37.204103 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.203633 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" Jul 08 20:29:37.204103 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.203677 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" event={"ID":"25fca6b2-34d0-450f-8317-2d175b5b293f","Type":"ContainerDied","Data":"042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3"} Jul 08 20:29:37.204103 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.203740 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6" event={"ID":"25fca6b2-34d0-450f-8317-2d175b5b293f","Type":"ContainerDied","Data":"15213d6aa178e47e7d3f0002918c25f6a1cd2cac1ed22432fff7f64102b41e84"} Jul 08 20:29:37.204103 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.203776 2573 scope.go:117] "RemoveContainer" containerID="042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3" Jul 08 20:29:37.213464 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.213438 2573 scope.go:117] "RemoveContainer" containerID="042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3" Jul 08 20:29:37.213761 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:29:37.213740 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3\": container with ID starting with 042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3 not found: ID does not exist" containerID="042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3" Jul 08 20:29:37.213826 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.213774 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3"} err="failed to get container status \"042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3\": rpc error: code = NotFound desc = could not find container \"042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3\": container with ID starting with 042ba3e7c039fc2d2e5bbf955a518432c10f2bf4405212d715c6fbc21e62dcd3 not found: ID does not exist" Jul 08 20:29:37.226652 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.226624 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6"] Jul 08 20:29:37.228424 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.228400 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-76d496a5-85fd456cd7-ws5n6"] Jul 08 20:29:37.240320 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.240289 2573 patch_prober.go:28] interesting pod/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 08 20:29:37.240399 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:37.240354 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 08 20:29:38.111714 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:38.111672 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:38.239661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:38.239624 2573 patch_prober.go:28] interesting pod/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 08 20:29:38.240067 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:38.239695 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 08 20:29:38.787352 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:38.787298 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25fca6b2-34d0-450f-8317-2d175b5b293f" path="/var/lib/kubelet/pods/25fca6b2-34d0-450f-8317-2d175b5b293f/volumes" Jul 08 20:29:39.240232 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:39.240198 2573 patch_prober.go:28] interesting pod/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 08 20:29:39.240636 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:39.240252 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 08 20:29:40.245023 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:40.244997 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:40.246037 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:40.246016 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:42.281691 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:42.281653 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:42.381732 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:42.381696 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:42.537344 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:42.537232 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq"] Jul 08 20:29:42.537510 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:42.537477 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" containerID="cri-o://f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717" gracePeriod=30 Jul 08 20:29:42.566177 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:42.566059 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:42.928101 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:42.928076 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:43.030803 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:43.030774 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:43.176167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:43.176133 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:47.790332 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.790292 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:47.873016 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.872931 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-istio-token\") pod \"20ba3330-29e6-439e-af48-739177845682\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " Jul 08 20:29:47.873176 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873017 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-data\") pod \"20ba3330-29e6-439e-af48-739177845682\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " Jul 08 20:29:47.873176 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873056 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-credential-socket\") pod \"20ba3330-29e6-439e-af48-739177845682\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " Jul 08 20:29:47.873176 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873080 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-socket\") pod \"20ba3330-29e6-439e-af48-739177845682\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " Jul 08 20:29:47.873176 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873105 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-certs\") pod \"20ba3330-29e6-439e-af48-739177845682\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " Jul 08 20:29:47.873176 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873144 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-envoy\") pod \"20ba3330-29e6-439e-af48-739177845682\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " Jul 08 20:29:47.873176 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873170 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86qth\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-kube-api-access-86qth\") pod \"20ba3330-29e6-439e-af48-739177845682\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " Jul 08 20:29:47.873532 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873197 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ba3330-29e6-439e-af48-739177845682-istiod-ca-cert\") pod \"20ba3330-29e6-439e-af48-739177845682\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " Jul 08 20:29:47.873532 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873244 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ba3330-29e6-439e-af48-739177845682-istio-podinfo\") pod \"20ba3330-29e6-439e-af48-739177845682\" (UID: \"20ba3330-29e6-439e-af48-739177845682\") " Jul 08 20:29:47.873532 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873345 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "20ba3330-29e6-439e-af48-739177845682" (UID: "20ba3330-29e6-439e-af48-739177845682"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:29:47.873690 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873591 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:47.873690 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873656 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "20ba3330-29e6-439e-af48-739177845682" (UID: "20ba3330-29e6-439e-af48-739177845682"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:29:47.873690 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873670 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "20ba3330-29e6-439e-af48-739177845682" (UID: "20ba3330-29e6-439e-af48-739177845682"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:29:47.873806 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873705 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ba3330-29e6-439e-af48-739177845682-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "20ba3330-29e6-439e-af48-739177845682" (UID: "20ba3330-29e6-439e-af48-739177845682"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:29:47.873891 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.873867 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-data" (OuterVolumeSpecName: "istio-data") pod "20ba3330-29e6-439e-af48-739177845682" (UID: "20ba3330-29e6-439e-af48-739177845682"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:29:47.875163 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.875143 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-istio-token" (OuterVolumeSpecName: "istio-token") pod "20ba3330-29e6-439e-af48-739177845682" (UID: "20ba3330-29e6-439e-af48-739177845682"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:29:47.875458 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.875433 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-kube-api-access-86qth" (OuterVolumeSpecName: "kube-api-access-86qth") pod "20ba3330-29e6-439e-af48-739177845682" (UID: "20ba3330-29e6-439e-af48-739177845682"). InnerVolumeSpecName "kube-api-access-86qth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:29:47.875656 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.875630 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "20ba3330-29e6-439e-af48-739177845682" (UID: "20ba3330-29e6-439e-af48-739177845682"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:29:47.875742 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.875704 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20ba3330-29e6-439e-af48-739177845682-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "20ba3330-29e6-439e-af48-739177845682" (UID: "20ba3330-29e6-439e-af48-739177845682"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:29:47.974364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.974333 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:47.974364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.974359 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86qth\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-kube-api-access-86qth\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:47.974364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.974369 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ba3330-29e6-439e-af48-739177845682-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:47.974597 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.974378 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ba3330-29e6-439e-af48-739177845682-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:47.974597 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.974386 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ba3330-29e6-439e-af48-739177845682-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:47.974597 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.974393 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:47.974597 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.974400 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:47.974597 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:47.974407 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ba3330-29e6-439e-af48-739177845682-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:29:48.262168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.262134 2573 generic.go:356] "Generic (PLEG): container finished" podID="20ba3330-29e6-439e-af48-739177845682" containerID="f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717" exitCode=0 Jul 08 20:29:48.262357 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.262181 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" Jul 08 20:29:48.262357 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.262181 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" event={"ID":"20ba3330-29e6-439e-af48-739177845682","Type":"ContainerDied","Data":"f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717"} Jul 08 20:29:48.262357 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.262294 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq" event={"ID":"20ba3330-29e6-439e-af48-739177845682","Type":"ContainerDied","Data":"6ea28dcb1031fbb44d66304cd667afe835fef3999ff457965ba108e9aa4695e1"} Jul 08 20:29:48.262357 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.262341 2573 scope.go:117] "RemoveContainer" containerID="f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717" Jul 08 20:29:48.271460 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.271441 2573 scope.go:117] "RemoveContainer" containerID="c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd" Jul 08 20:29:48.279814 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.279798 2573 scope.go:117] "RemoveContainer" containerID="f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717" Jul 08 20:29:48.280047 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:29:48.280031 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717\": container with ID starting with f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717 not found: ID does not exist" containerID="f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717" Jul 08 20:29:48.280088 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.280055 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717"} err="failed to get container status \"f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717\": rpc error: code = NotFound desc = could not find container \"f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717\": container with ID starting with f87926d7c213a7bd3b0d81b0b1a78239c5771d52962f9654dbdc9426a69a3717 not found: ID does not exist" Jul 08 20:29:48.280088 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.280073 2573 scope.go:117] "RemoveContainer" containerID="c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd" Jul 08 20:29:48.280268 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:29:48.280253 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd\": container with ID starting with c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd not found: ID does not exist" containerID="c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd" Jul 08 20:29:48.280320 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.280275 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd"} err="failed to get container status \"c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd\": rpc error: code = NotFound desc = could not find container \"c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd\": container with ID starting with c361160c890d03f08889677e4d60b54db6d72fa3c85ac139e5b0463be6581bbd not found: ID does not exist" Jul 08 20:29:48.283817 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.283798 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq"] Jul 08 20:29:48.287409 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.287387 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76d496a5-openshift-default-7545666b75-dmmfq"] Jul 08 20:29:48.787257 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.787224 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20ba3330-29e6-439e-af48-739177845682" path="/var/lib/kubelet/pods/20ba3330-29e6-439e-af48-739177845682/volumes" Jul 08 20:29:48.965533 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:48.965481 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:49.068401 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:49.068301 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:49.165949 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:49.165917 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:49.363666 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:49.363582 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:55.119169 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:55.119139 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:29:55.263954 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:29:55.263925 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:00.139269 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.139233 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg"] Jul 08 20:30:00.139699 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.139665 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fca6b2-34d0-450f-8317-2d175b5b293f" containerName="maas-api" Jul 08 20:30:00.139699 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.139679 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fca6b2-34d0-450f-8317-2d175b5b293f" containerName="maas-api" Jul 08 20:30:00.139699 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.139700 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" Jul 08 20:30:00.139811 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.139706 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" Jul 08 20:30:00.139811 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.139776 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fca6b2-34d0-450f-8317-2d175b5b293f" containerName="maas-api" Jul 08 20:30:00.139811 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.139788 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" Jul 08 20:30:00.139811 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.139798 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" Jul 08 20:30:00.144603 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.144584 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" Jul 08 20:30:00.148548 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.148341 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:30:00.148548 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.148365 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x22jf\"" Jul 08 20:30:00.148548 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.148350 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:30:00.149722 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.149696 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg"] Jul 08 20:30:00.282498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.282468 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd76g\" (UniqueName: \"kubernetes.io/projected/aeb8ebfc-a265-4513-9c23-4a2a3a389659-kube-api-access-dd76g\") pod \"maas-api-key-cleanup-29725710-phhvg\" (UID: \"aeb8ebfc-a265-4513-9c23-4a2a3a389659\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" Jul 08 20:30:00.383065 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.383043 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd76g\" (UniqueName: \"kubernetes.io/projected/aeb8ebfc-a265-4513-9c23-4a2a3a389659-kube-api-access-dd76g\") pod \"maas-api-key-cleanup-29725710-phhvg\" (UID: \"aeb8ebfc-a265-4513-9c23-4a2a3a389659\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" Jul 08 20:30:00.390797 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.390748 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd76g\" (UniqueName: \"kubernetes.io/projected/aeb8ebfc-a265-4513-9c23-4a2a3a389659-kube-api-access-dd76g\") pod \"maas-api-key-cleanup-29725710-phhvg\" (UID: \"aeb8ebfc-a265-4513-9c23-4a2a3a389659\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" Jul 08 20:30:00.458786 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.458768 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" Jul 08 20:30:00.586940 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:00.586921 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg"] Jul 08 20:30:00.588636 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:30:00.588614 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeb8ebfc_a265_4513_9c23_4a2a3a389659.slice/crio-d85b571dcc754e0761e69999b938611c48e5b23463570c153c157671f3c98d47 WatchSource:0}: Error finding container d85b571dcc754e0761e69999b938611c48e5b23463570c153c157671f3c98d47: Status 404 returned error can't find the container with id d85b571dcc754e0761e69999b938611c48e5b23463570c153c157671f3c98d47 Jul 08 20:30:01.326700 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:01.326663 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" event={"ID":"aeb8ebfc-a265-4513-9c23-4a2a3a389659","Type":"ContainerStarted","Data":"d85b571dcc754e0761e69999b938611c48e5b23463570c153c157671f3c98d47"} Jul 08 20:30:02.212951 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.212918 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:02.406768 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.406738 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:02.530626 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.530549 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs"] Jul 08 20:30:02.531169 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.531149 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" Jul 08 20:30:02.531267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.531172 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="20ba3330-29e6-439e-af48-739177845682" containerName="istio-proxy" Jul 08 20:30:02.540294 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.540267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.543080 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.543059 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5154d846-openshift-default-dockercfg-6v7gk\"" Jul 08 20:30:02.543329 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.543060 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 20:30:02.544295 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.544271 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs"] Jul 08 20:30:02.554564 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.554516 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:02.602767 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.602738 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-istio-token\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.602910 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.602821 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-certs\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.602910 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.602877 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50e6144-7c44-4f24-a746-b121d406a83f-istio-podinfo\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.602910 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.602904 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-socket\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.603080 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.602932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blzdn\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-kube-api-access-blzdn\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.603080 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.603036 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-data\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.603080 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.603074 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-envoy\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.603194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.603113 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-credential-socket\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.603194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.603139 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50e6144-7c44-4f24-a746-b121d406a83f-istiod-ca-cert\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704183 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704148 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-credential-socket\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704369 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50e6144-7c44-4f24-a746-b121d406a83f-istiod-ca-cert\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704369 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704257 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-istio-token\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704369 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704328 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-certs\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704541 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50e6144-7c44-4f24-a746-b121d406a83f-istio-podinfo\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704541 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704415 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-socket\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704541 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704452 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blzdn\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-kube-api-access-blzdn\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704541 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-data\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704541 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704523 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-envoy\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.704774 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704587 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-credential-socket\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.705009 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704957 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50e6144-7c44-4f24-a746-b121d406a83f-istiod-ca-cert\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.705009 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.704981 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-certs\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.705179 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.705111 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-socket\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.705224 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.705200 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-data\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.707051 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.707027 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-envoy\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.707211 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.707190 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50e6144-7c44-4f24-a746-b121d406a83f-istio-podinfo\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.712478 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.712455 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-istio-token\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.712964 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.712943 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blzdn\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-kube-api-access-blzdn\") pod \"e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:02.855186 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:02.855110 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:03.099102 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:03.099079 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs"] Jul 08 20:30:03.101879 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:30:03.101858 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50e6144_7c44_4f24_a746_b121d406a83f.slice/crio-5ed976b90c5fb4a5a9b9e269a85a7f663e788b0e2a3e61da3762b21dac579018 WatchSource:0}: Error finding container 5ed976b90c5fb4a5a9b9e269a85a7f663e788b0e2a3e61da3762b21dac579018: Status 404 returned error can't find the container with id 5ed976b90c5fb4a5a9b9e269a85a7f663e788b0e2a3e61da3762b21dac579018 Jul 08 20:30:03.104130 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:03.104085 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:30:03.104236 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:03.104166 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:30:03.104236 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:03.104204 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:30:03.330743 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:03.330716 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:03.339668 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:03.339644 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" event={"ID":"d50e6144-7c44-4f24-a746-b121d406a83f","Type":"ContainerStarted","Data":"f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f"} Jul 08 20:30:03.339805 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:03.339691 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" event={"ID":"d50e6144-7c44-4f24-a746-b121d406a83f","Type":"ContainerStarted","Data":"5ed976b90c5fb4a5a9b9e269a85a7f663e788b0e2a3e61da3762b21dac579018"} Jul 08 20:30:03.362166 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:03.362100 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" podStartSLOduration=1.362084722 podStartE2EDuration="1.362084722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:30:03.359649145 +0000 UTC m=+1771.356666092" watchObservedRunningTime="2026-07-08 20:30:03.362084722 +0000 UTC m=+1771.359101657" Jul 08 20:30:03.855986 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:03.855959 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:04.345915 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:04.345884 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" event={"ID":"aeb8ebfc-a265-4513-9c23-4a2a3a389659","Type":"ContainerStarted","Data":"fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c"} Jul 08 20:30:04.361483 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:04.361438 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" podStartSLOduration=1.909304318 podStartE2EDuration="4.361423462s" totalImagesPullingTime="2.452119144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:30:04.360519485 +0000 UTC m=+1772.357536419" watchObservedRunningTime="2026-07-08 20:30:04.361423462 +0000 UTC m=+1772.358440396" Jul 08 20:30:04.860554 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:04.860529 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:05.338513 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.338481 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:05.351589 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.351555 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:05.352530 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.352508 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:05.437850 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.437824 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:05.584090 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.584065 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2"] Jul 08 20:30:05.587768 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.587749 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:05.590564 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.590506 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5154d846-serving-cert\"" Jul 08 20:30:05.596086 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.596059 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2"] Jul 08 20:30:05.628672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.628648 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:05.731772 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.731750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193a565c-6299-431e-952f-a69ba229f972-maas-api-tls\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:05.731881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.731788 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzvl\" (UniqueName: \"kubernetes.io/projected/193a565c-6299-431e-952f-a69ba229f972-kube-api-access-srzvl\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:05.731881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.731870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193a565c-6299-431e-952f-a69ba229f972-openshift-service-ca\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:05.833125 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.833102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193a565c-6299-431e-952f-a69ba229f972-maas-api-tls\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:05.833208 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.833142 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzvl\" (UniqueName: \"kubernetes.io/projected/193a565c-6299-431e-952f-a69ba229f972-kube-api-access-srzvl\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:05.833208 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.833188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193a565c-6299-431e-952f-a69ba229f972-openshift-service-ca\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:05.833296 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:30:05.833239 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-serving-cert: secret "maas-api-e2e-mt-5154d846-serving-cert" not found Jul 08 20:30:05.833383 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:30:05.833364 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/193a565c-6299-431e-952f-a69ba229f972-maas-api-tls podName:193a565c-6299-431e-952f-a69ba229f972 nodeName:}" failed. No retries permitted until 2026-07-08 20:30:06.333280885 +0000 UTC m=+1774.330297798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/193a565c-6299-431e-952f-a69ba229f972-maas-api-tls") pod "maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" (UID: "193a565c-6299-431e-952f-a69ba229f972") : secret "maas-api-e2e-mt-5154d846-serving-cert" not found Jul 08 20:30:05.833926 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.833905 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193a565c-6299-431e-952f-a69ba229f972-openshift-service-ca\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:05.842327 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:05.842263 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzvl\" (UniqueName: \"kubernetes.io/projected/193a565c-6299-431e-952f-a69ba229f972-kube-api-access-srzvl\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:06.337185 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:06.337153 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193a565c-6299-431e-952f-a69ba229f972-maas-api-tls\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:06.339614 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:06.339582 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193a565c-6299-431e-952f-a69ba229f972-maas-api-tls\") pod \"maas-api-e2e-mt-5154d846-6d648d6476-m2bp2\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:06.502071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:06.502041 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:06.600355 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:06.600229 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:06.635539 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:06.635518 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2"] Jul 08 20:30:06.637788 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:30:06.637766 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193a565c_6299_431e_952f_a69ba229f972.slice/crio-7ed91d6a1e0545c95beec0fff45702733ba72d735f7c4e4a213108ba2a9b7561 WatchSource:0}: Error finding container 7ed91d6a1e0545c95beec0fff45702733ba72d735f7c4e4a213108ba2a9b7561: Status 404 returned error can't find the container with id 7ed91d6a1e0545c95beec0fff45702733ba72d735f7c4e4a213108ba2a9b7561 Jul 08 20:30:06.700380 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:06.700356 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:07.362588 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:07.362500 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" event={"ID":"193a565c-6299-431e-952f-a69ba229f972","Type":"ContainerStarted","Data":"690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e"} Jul 08 20:30:07.362588 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:07.362552 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" event={"ID":"193a565c-6299-431e-952f-a69ba229f972","Type":"ContainerStarted","Data":"7ed91d6a1e0545c95beec0fff45702733ba72d735f7c4e4a213108ba2a9b7561"} Jul 08 20:30:07.363078 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:07.362698 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:07.380387 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:07.380340 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" podStartSLOduration=2.001618608 podStartE2EDuration="2.380323955s" totalImagesPullingTime="378.705347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:30:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:30:07.379108521 +0000 UTC m=+1775.376125455" watchObservedRunningTime="2026-07-08 20:30:07.380323955 +0000 UTC m=+1775.377340889" Jul 08 20:30:09.925107 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:09.925073 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:13.373612 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:13.373587 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:13.486273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:13.486242 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:13.588134 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:13.588104 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:19.426954 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:19.426918 2573 generic.go:356] "Generic (PLEG): container finished" podID="d50e6144-7c44-4f24-a746-b121d406a83f" containerID="f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f" exitCode=0 Jul 08 20:30:19.427389 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:19.426967 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" event={"ID":"d50e6144-7c44-4f24-a746-b121d406a83f","Type":"ContainerDied","Data":"f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f"} Jul 08 20:30:19.427389 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:19.427355 2573 scope.go:117] "RemoveContainer" containerID="f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f" Jul 08 20:30:19.428404 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:19.428365 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:30:19.428523 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:19.428455 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:30:19.428523 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:19.428504 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:30:20.351644 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:20.351600 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:20.433909 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:20.433877 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" event={"ID":"d50e6144-7c44-4f24-a746-b121d406a83f","Type":"ContainerStarted","Data":"c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b"} Jul 08 20:30:20.601969 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:20.601893 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:20.856281 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:20.856193 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:20.857562 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:20.857531 2573 patch_prober.go:28] interesting pod/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 08 20:30:20.857676 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:20.857599 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 08 20:30:21.855851 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:21.855806 2573 patch_prober.go:28] interesting pod/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 08 20:30:21.856403 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:21.855878 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 08 20:30:22.856152 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:22.856113 2573 patch_prober.go:28] interesting pod/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 08 20:30:22.856667 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:22.856185 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 08 20:30:23.856091 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:23.856059 2573 patch_prober.go:28] interesting pod/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 08 20:30:23.856276 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:23.856124 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 08 20:30:24.859455 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:24.859420 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:24.859992 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:24.859794 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:24.860445 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:24.860426 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:30:27.822476 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:27.822438 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:27.921611 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:27.921584 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:32.680182 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:32.680153 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:30:32.685268 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:32.685247 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:30:42.528904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:42.528873 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:42.913593 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:42.913497 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2"] Jul 08 20:30:42.913780 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:42.913733 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" podUID="193a565c-6299-431e-952f-a69ba229f972" containerName="maas-api" containerID="cri-o://690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e" gracePeriod=30 Jul 08 20:30:42.941178 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:42.941155 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:43.170053 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.170034 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:43.270213 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.270192 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srzvl\" (UniqueName: \"kubernetes.io/projected/193a565c-6299-431e-952f-a69ba229f972-kube-api-access-srzvl\") pod \"193a565c-6299-431e-952f-a69ba229f972\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " Jul 08 20:30:43.270359 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.270220 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193a565c-6299-431e-952f-a69ba229f972-maas-api-tls\") pod \"193a565c-6299-431e-952f-a69ba229f972\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " Jul 08 20:30:43.270359 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.270329 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193a565c-6299-431e-952f-a69ba229f972-openshift-service-ca\") pod \"193a565c-6299-431e-952f-a69ba229f972\" (UID: \"193a565c-6299-431e-952f-a69ba229f972\") " Jul 08 20:30:43.270701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.270677 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/193a565c-6299-431e-952f-a69ba229f972-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "193a565c-6299-431e-952f-a69ba229f972" (UID: "193a565c-6299-431e-952f-a69ba229f972"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:30:43.272233 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.272207 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/193a565c-6299-431e-952f-a69ba229f972-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "193a565c-6299-431e-952f-a69ba229f972" (UID: "193a565c-6299-431e-952f-a69ba229f972"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:30:43.272352 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.272333 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/193a565c-6299-431e-952f-a69ba229f972-kube-api-access-srzvl" (OuterVolumeSpecName: "kube-api-access-srzvl") pod "193a565c-6299-431e-952f-a69ba229f972" (UID: "193a565c-6299-431e-952f-a69ba229f972"). InnerVolumeSpecName "kube-api-access-srzvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:30:43.371518 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.371498 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/193a565c-6299-431e-952f-a69ba229f972-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:30:43.371518 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.371519 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzvl\" (UniqueName: \"kubernetes.io/projected/193a565c-6299-431e-952f-a69ba229f972-kube-api-access-srzvl\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:30:43.371644 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.371528 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/193a565c-6299-431e-952f-a69ba229f972-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:30:43.554791 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.554738 2573 generic.go:356] "Generic (PLEG): container finished" podID="193a565c-6299-431e-952f-a69ba229f972" containerID="690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e" exitCode=0 Jul 08 20:30:43.554791 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.554780 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" Jul 08 20:30:43.555145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.554790 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" event={"ID":"193a565c-6299-431e-952f-a69ba229f972","Type":"ContainerDied","Data":"690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e"} Jul 08 20:30:43.555145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.554835 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2" event={"ID":"193a565c-6299-431e-952f-a69ba229f972","Type":"ContainerDied","Data":"7ed91d6a1e0545c95beec0fff45702733ba72d735f7c4e4a213108ba2a9b7561"} Jul 08 20:30:43.555145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.554856 2573 scope.go:117] "RemoveContainer" containerID="690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e" Jul 08 20:30:43.563517 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.563500 2573 scope.go:117] "RemoveContainer" containerID="690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e" Jul 08 20:30:43.563775 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:30:43.563759 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e\": container with ID starting with 690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e not found: ID does not exist" containerID="690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e" Jul 08 20:30:43.563814 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.563782 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e"} err="failed to get container status \"690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e\": rpc error: code = NotFound desc = could not find container \"690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e\": container with ID starting with 690d407619527dd3fd3a50fb85def3e9b58361e7fa3caa1e386259bd9496713e not found: ID does not exist" Jul 08 20:30:43.575783 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.575761 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2"] Jul 08 20:30:43.579994 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:43.579975 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5154d846-6d648d6476-m2bp2"] Jul 08 20:30:44.786423 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:44.786391 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="193a565c-6299-431e-952f-a69ba229f972" path="/var/lib/kubelet/pods/193a565c-6299-431e-952f-a69ba229f972/volumes" Jul 08 20:30:48.703705 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:48.703667 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:30:53.727213 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:30:53.727182 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:00.011035 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:00.011001 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg"] Jul 08 20:31:00.011448 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:00.011182 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" podUID="aeb8ebfc-a265-4513-9c23-4a2a3a389659" containerName="cleanup" containerID="cri-o://fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c" gracePeriod=30 Jul 08 20:31:26.953098 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:26.953067 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:27.426745 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:27.426715 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:30.193582 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.193555 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" Jul 08 20:31:30.274831 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.274763 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd76g\" (UniqueName: \"kubernetes.io/projected/aeb8ebfc-a265-4513-9c23-4a2a3a389659-kube-api-access-dd76g\") pod \"aeb8ebfc-a265-4513-9c23-4a2a3a389659\" (UID: \"aeb8ebfc-a265-4513-9c23-4a2a3a389659\") " Jul 08 20:31:30.276651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.276624 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb8ebfc-a265-4513-9c23-4a2a3a389659-kube-api-access-dd76g" (OuterVolumeSpecName: "kube-api-access-dd76g") pod "aeb8ebfc-a265-4513-9c23-4a2a3a389659" (UID: "aeb8ebfc-a265-4513-9c23-4a2a3a389659"). InnerVolumeSpecName "kube-api-access-dd76g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:31:30.375761 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.375737 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd76g\" (UniqueName: \"kubernetes.io/projected/aeb8ebfc-a265-4513-9c23-4a2a3a389659-kube-api-access-dd76g\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:30.799036 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.799010 2573 generic.go:356] "Generic (PLEG): container finished" podID="aeb8ebfc-a265-4513-9c23-4a2a3a389659" containerID="fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c" exitCode=137 Jul 08 20:31:30.799164 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.799049 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" event={"ID":"aeb8ebfc-a265-4513-9c23-4a2a3a389659","Type":"ContainerDied","Data":"fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c"} Jul 08 20:31:30.799164 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.799052 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" Jul 08 20:31:30.799164 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.799092 2573 scope.go:117] "RemoveContainer" containerID="fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c" Jul 08 20:31:30.799329 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.799082 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg" event={"ID":"aeb8ebfc-a265-4513-9c23-4a2a3a389659","Type":"ContainerDied","Data":"d85b571dcc754e0761e69999b938611c48e5b23463570c153c157671f3c98d47"} Jul 08 20:31:30.810147 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.810110 2573 scope.go:117] "RemoveContainer" containerID="fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c" Jul 08 20:31:30.810543 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:31:30.810520 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c\": container with ID starting with fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c not found: ID does not exist" containerID="fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c" Jul 08 20:31:30.810641 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.810553 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c"} err="failed to get container status \"fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c\": rpc error: code = NotFound desc = could not find container \"fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c\": container with ID starting with fe9a87e613df5bd5ca28b3f4361351eb5cdb8e99056f9b1bddb273457b4bbe0c not found: ID does not exist" Jul 08 20:31:30.815909 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.815890 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg"] Jul 08 20:31:30.817656 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:30.817636 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725710-phhvg"] Jul 08 20:31:32.786747 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:32.786718 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb8ebfc-a265-4513-9c23-4a2a3a389659" path="/var/lib/kubelet/pods/aeb8ebfc-a265-4513-9c23-4a2a3a389659/volumes" Jul 08 20:31:33.200496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:33.200464 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:38.754108 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:38.754072 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:38.857238 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:38.857212 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:39.028651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:39.028567 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs"] Jul 08 20:31:39.028873 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:39.028847 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" containerID="cri-o://c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b" gracePeriod=30 Jul 08 20:31:39.047002 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:39.046978 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:39.862328 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:39.862275 2573 patch_prober.go:28] interesting pod/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 20:31:39.862720 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:39.862373 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 20:31:40.427564 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:40.427534 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:40.529359 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:40.529304 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:40.629223 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:40.629201 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:41.438797 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:41.438771 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:41.539076 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:41.539046 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:41.652288 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:41.652260 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:43.223354 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:43.223296 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:44.268475 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.268455 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:31:44.395789 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.395738 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-envoy\") pod \"d50e6144-7c44-4f24-a746-b121d406a83f\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " Jul 08 20:31:44.395789 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.395763 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50e6144-7c44-4f24-a746-b121d406a83f-istiod-ca-cert\") pod \"d50e6144-7c44-4f24-a746-b121d406a83f\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " Jul 08 20:31:44.395789 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.395787 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-data\") pod \"d50e6144-7c44-4f24-a746-b121d406a83f\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " Jul 08 20:31:44.395951 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.395803 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-credential-socket\") pod \"d50e6144-7c44-4f24-a746-b121d406a83f\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " Jul 08 20:31:44.395992 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.395943 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blzdn\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-kube-api-access-blzdn\") pod \"d50e6144-7c44-4f24-a746-b121d406a83f\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " Jul 08 20:31:44.396042 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.395991 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-istio-token\") pod \"d50e6144-7c44-4f24-a746-b121d406a83f\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " Jul 08 20:31:44.396094 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396044 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-certs\") pod \"d50e6144-7c44-4f24-a746-b121d406a83f\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " Jul 08 20:31:44.396146 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396123 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d50e6144-7c44-4f24-a746-b121d406a83f" (UID: "d50e6144-7c44-4f24-a746-b121d406a83f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:31:44.396239 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396178 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50e6144-7c44-4f24-a746-b121d406a83f-istio-podinfo\") pod \"d50e6144-7c44-4f24-a746-b121d406a83f\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " Jul 08 20:31:44.396378 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396215 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-socket\") pod \"d50e6144-7c44-4f24-a746-b121d406a83f\" (UID: \"d50e6144-7c44-4f24-a746-b121d406a83f\") " Jul 08 20:31:44.396378 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396291 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d50e6144-7c44-4f24-a746-b121d406a83f" (UID: "d50e6144-7c44-4f24-a746-b121d406a83f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:31:44.396468 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396387 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d50e6144-7c44-4f24-a746-b121d406a83f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d50e6144-7c44-4f24-a746-b121d406a83f" (UID: "d50e6144-7c44-4f24-a746-b121d406a83f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:31:44.396601 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396573 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d50e6144-7c44-4f24-a746-b121d406a83f" (UID: "d50e6144-7c44-4f24-a746-b121d406a83f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:31:44.396713 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396620 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-data" (OuterVolumeSpecName: "istio-data") pod "d50e6144-7c44-4f24-a746-b121d406a83f" (UID: "d50e6144-7c44-4f24-a746-b121d406a83f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:31:44.396713 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396696 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:44.396796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396714 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:44.396796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396728 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d50e6144-7c44-4f24-a746-b121d406a83f-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:44.396796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.396742 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:44.398043 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.398021 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-kube-api-access-blzdn" (OuterVolumeSpecName: "kube-api-access-blzdn") pod "d50e6144-7c44-4f24-a746-b121d406a83f" (UID: "d50e6144-7c44-4f24-a746-b121d406a83f"). InnerVolumeSpecName "kube-api-access-blzdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:31:44.398182 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.398158 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d50e6144-7c44-4f24-a746-b121d406a83f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d50e6144-7c44-4f24-a746-b121d406a83f" (UID: "d50e6144-7c44-4f24-a746-b121d406a83f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:31:44.398267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.398247 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d50e6144-7c44-4f24-a746-b121d406a83f" (UID: "d50e6144-7c44-4f24-a746-b121d406a83f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:31:44.398596 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.398577 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d50e6144-7c44-4f24-a746-b121d406a83f" (UID: "d50e6144-7c44-4f24-a746-b121d406a83f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:31:44.497093 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.497073 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:44.497184 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.497095 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blzdn\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-kube-api-access-blzdn\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:44.497184 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.497109 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d50e6144-7c44-4f24-a746-b121d406a83f-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:44.497184 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.497120 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d50e6144-7c44-4f24-a746-b121d406a83f-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:44.497184 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.497135 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d50e6144-7c44-4f24-a746-b121d406a83f-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:31:44.870762 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.870734 2573 generic.go:356] "Generic (PLEG): container finished" podID="d50e6144-7c44-4f24-a746-b121d406a83f" containerID="c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b" exitCode=0 Jul 08 20:31:44.870881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.870787 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" Jul 08 20:31:44.870881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.870817 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" event={"ID":"d50e6144-7c44-4f24-a746-b121d406a83f","Type":"ContainerDied","Data":"c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b"} Jul 08 20:31:44.870881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.870871 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs" event={"ID":"d50e6144-7c44-4f24-a746-b121d406a83f","Type":"ContainerDied","Data":"5ed976b90c5fb4a5a9b9e269a85a7f663e788b0e2a3e61da3762b21dac579018"} Jul 08 20:31:44.871011 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.870896 2573 scope.go:117] "RemoveContainer" containerID="c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b" Jul 08 20:31:44.880208 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.880193 2573 scope.go:117] "RemoveContainer" containerID="f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f" Jul 08 20:31:44.888145 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.888128 2573 scope.go:117] "RemoveContainer" containerID="c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b" Jul 08 20:31:44.888458 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:31:44.888438 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b\": container with ID starting with c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b not found: ID does not exist" containerID="c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b" Jul 08 20:31:44.888538 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.888470 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b"} err="failed to get container status \"c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b\": rpc error: code = NotFound desc = could not find container \"c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b\": container with ID starting with c7e56654cf19000b9d2188ea3946fe35e56f96f49edb0de54b67c6e659293e4b not found: ID does not exist" Jul 08 20:31:44.888538 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.888494 2573 scope.go:117] "RemoveContainer" containerID="f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f" Jul 08 20:31:44.888737 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:31:44.888719 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f\": container with ID starting with f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f not found: ID does not exist" containerID="f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f" Jul 08 20:31:44.888789 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.888744 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f"} err="failed to get container status \"f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f\": rpc error: code = NotFound desc = could not find container \"f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f\": container with ID starting with f349a0e4050b2bc529f43ee0cc518e2b8ef34364754cea7f00ceffe06446da4f not found: ID does not exist" Jul 08 20:31:44.894174 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.894150 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs"] Jul 08 20:31:44.897535 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:44.897514 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5154d846-openshift-default-6f69d7f67f-v95qs"] Jul 08 20:31:46.787723 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:46.787687 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" path="/var/lib/kubelet/pods/d50e6144-7c44-4f24-a746-b121d406a83f/volumes" Jul 08 20:31:49.354954 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:49.354925 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:49.454990 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:49.454953 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:50.078629 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:50.078602 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:55.749959 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:55.749875 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:55.810411 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:55.810385 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:55.886842 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:55.886818 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:31:55.994033 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:31:55.994005 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:06.553289 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:06.553254 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:06.932632 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:06.932604 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:27.971036 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:27.970998 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:33.751199 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:33.751166 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:33.817652 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:33.817622 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:34.137891 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.137798 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:34.451129 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451102 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx"] Jul 08 20:32:34.451512 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451499 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" Jul 08 20:32:34.451572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451514 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" Jul 08 20:32:34.451572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451526 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" Jul 08 20:32:34.451572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451531 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" Jul 08 20:32:34.451572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451538 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="193a565c-6299-431e-952f-a69ba229f972" containerName="maas-api" Jul 08 20:32:34.451572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451543 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="193a565c-6299-431e-952f-a69ba229f972" containerName="maas-api" Jul 08 20:32:34.451572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451561 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb8ebfc-a265-4513-9c23-4a2a3a389659" containerName="cleanup" Jul 08 20:32:34.451572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451567 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb8ebfc-a265-4513-9c23-4a2a3a389659" containerName="cleanup" Jul 08 20:32:34.451807 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451625 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="193a565c-6299-431e-952f-a69ba229f972" containerName="maas-api" Jul 08 20:32:34.451807 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451632 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" Jul 08 20:32:34.451807 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451641 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50e6144-7c44-4f24-a746-b121d406a83f" containerName="istio-proxy" Jul 08 20:32:34.451807 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.451648 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb8ebfc-a265-4513-9c23-4a2a3a389659" containerName="cleanup" Jul 08 20:32:34.456205 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.456189 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.459014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.458993 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-930853-openshift-default-dockercfg-xjgrr\"" Jul 08 20:32:34.459014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.459007 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 20:32:34.464968 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.464945 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx"] Jul 08 20:32:34.468836 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.468812 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:34.513722 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.513696 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-envoy\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.513824 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.513736 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstcd\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-kube-api-access-gstcd\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.513824 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.513760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-data\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.513898 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.513846 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-podinfo\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.513898 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.513892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-socket\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.513964 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.513943 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-credential-socket\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.513997 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.513968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-certs\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.514031 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.513995 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-token\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.514062 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.514030 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istiod-ca-cert\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.614584 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.614559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-envoy\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.614689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.614590 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstcd\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-kube-api-access-gstcd\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.614689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.614610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-data\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.614760 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.614716 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-podinfo\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.614797 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.614767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-socket\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.614849 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.614833 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-credential-socket\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.614892 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.614865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-certs\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.614932 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.614909 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-token\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.614978 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.614959 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istiod-ca-cert\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.615062 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.615042 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-data\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.615122 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.615084 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-socket\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.615250 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.615223 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-credential-socket\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.615389 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.615268 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-certs\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.615650 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.615629 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istiod-ca-cert\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.616705 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.616688 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-envoy\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.617106 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.617090 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-podinfo\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.622407 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.622386 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-token\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.622572 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.622554 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstcd\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-kube-api-access-gstcd\") pod \"e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.766799 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.766734 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:34.896366 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.896339 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx"] Jul 08 20:32:34.898545 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:32:34.898523 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205ac82a_139d_48c4_bc2a_bb8f1611ace8.slice/crio-e94bbc6df476e52c57d160d37f61e542f8b0049ef3fdd0e8ffb598ef9b5b327b WatchSource:0}: Error finding container e94bbc6df476e52c57d160d37f61e542f8b0049ef3fdd0e8ffb598ef9b5b327b: Status 404 returned error can't find the container with id e94bbc6df476e52c57d160d37f61e542f8b0049ef3fdd0e8ffb598ef9b5b327b Jul 08 20:32:34.900561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.900533 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:34.900664 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.900606 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:34.900664 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:34.900650 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:35.005647 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:35.005614 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:35.105693 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:35.105611 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" event={"ID":"205ac82a-139d-48c4-bc2a-bb8f1611ace8","Type":"ContainerStarted","Data":"f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0"} Jul 08 20:32:35.105693 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:35.105674 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" event={"ID":"205ac82a-139d-48c4-bc2a-bb8f1611ace8","Type":"ContainerStarted","Data":"e94bbc6df476e52c57d160d37f61e542f8b0049ef3fdd0e8ffb598ef9b5b327b"} Jul 08 20:32:35.127233 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:35.127168 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" podStartSLOduration=1.127151798 podStartE2EDuration="1.127151798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:32:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:32:35.125859228 +0000 UTC m=+1923.122876163" watchObservedRunningTime="2026-07-08 20:32:35.127151798 +0000 UTC m=+1923.124168803" Jul 08 20:32:35.766912 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:35.766883 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:35.771267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:35.771246 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:36.111623 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:36.111536 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:36.112467 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:36.112446 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:37.108460 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.108419 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:37.189481 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.189431 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:37.194986 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.194958 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t"] Jul 08 20:32:37.198580 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.198557 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.201685 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.201665 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-930853-serving-cert\"" Jul 08 20:32:37.201808 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.201668 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:32:37.202906 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.202887 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x22jf\"" Jul 08 20:32:37.203000 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.202889 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:32:37.210857 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.210830 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t"] Jul 08 20:32:37.293261 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.293230 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:37.340826 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.340793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7w87\" (UniqueName: \"kubernetes.io/projected/e2e54471-5158-48a2-b643-0d0c7153ba03-kube-api-access-f7w87\") pod \"maas-api-e2e-inf-a-930853-f7fff676f-d5c7t\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.341039 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.340850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e54471-5158-48a2-b643-0d0c7153ba03-maas-api-tls\") pod \"maas-api-e2e-inf-a-930853-f7fff676f-d5c7t\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.341039 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.340930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e54471-5158-48a2-b643-0d0c7153ba03-openshift-service-ca\") pod \"maas-api-e2e-inf-a-930853-f7fff676f-d5c7t\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.390751 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.390667 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:37.442268 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.442231 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e54471-5158-48a2-b643-0d0c7153ba03-maas-api-tls\") pod \"maas-api-e2e-inf-a-930853-f7fff676f-d5c7t\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.442475 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.442279 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e54471-5158-48a2-b643-0d0c7153ba03-openshift-service-ca\") pod \"maas-api-e2e-inf-a-930853-f7fff676f-d5c7t\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.442475 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.442433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7w87\" (UniqueName: \"kubernetes.io/projected/e2e54471-5158-48a2-b643-0d0c7153ba03-kube-api-access-f7w87\") pod \"maas-api-e2e-inf-a-930853-f7fff676f-d5c7t\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.443047 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.443014 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e54471-5158-48a2-b643-0d0c7153ba03-openshift-service-ca\") pod \"maas-api-e2e-inf-a-930853-f7fff676f-d5c7t\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.444624 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.444606 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e54471-5158-48a2-b643-0d0c7153ba03-maas-api-tls\") pod \"maas-api-e2e-inf-a-930853-f7fff676f-d5c7t\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.450556 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.450525 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7w87\" (UniqueName: \"kubernetes.io/projected/e2e54471-5158-48a2-b643-0d0c7153ba03-kube-api-access-f7w87\") pod \"maas-api-e2e-inf-a-930853-f7fff676f-d5c7t\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.512782 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.512742 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:37.644203 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.644178 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t"] Jul 08 20:32:37.645642 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:32:37.645613 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e54471_5158_48a2_b643_0d0c7153ba03.slice/crio-8bd81cd4c6809f942c7858edae8b9cce886b7f623234cf1d95be2bd1fc2e0d09 WatchSource:0}: Error finding container 8bd81cd4c6809f942c7858edae8b9cce886b7f623234cf1d95be2bd1fc2e0d09: Status 404 returned error can't find the container with id 8bd81cd4c6809f942c7858edae8b9cce886b7f623234cf1d95be2bd1fc2e0d09 Jul 08 20:32:37.647053 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.647033 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:32:37.767224 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.767191 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:37.875224 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:37.875195 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:38.126499 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:38.126467 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" event={"ID":"e2e54471-5158-48a2-b643-0d0c7153ba03","Type":"ContainerStarted","Data":"8bd81cd4c6809f942c7858edae8b9cce886b7f623234cf1d95be2bd1fc2e0d09"} Jul 08 20:32:39.132957 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:39.132917 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" event={"ID":"e2e54471-5158-48a2-b643-0d0c7153ba03","Type":"ContainerStarted","Data":"0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a"} Jul 08 20:32:39.133366 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:39.133036 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:39.150203 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:39.150151 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" podStartSLOduration=1.626027968 podStartE2EDuration="2.150134971s" totalImagesPullingTime="524.107003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:32:39.148197604 +0000 UTC m=+1927.145214554" watchObservedRunningTime="2026-07-08 20:32:39.150134971 +0000 UTC m=+1927.147151908" Jul 08 20:32:41.701479 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:41.701447 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:42.078053 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.077970 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:42.416657 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.416625 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7"] Jul 08 20:32:42.420257 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.420241 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.423242 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.423220 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-879445-openshift-default-dockercfg-5zbfv\"" Jul 08 20:32:42.434570 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.434547 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7"] Jul 08 20:32:42.467177 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.467144 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:42.492335 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.492283 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97892a82-4363-435f-b8ac-ea2f866a25ec-istiod-ca-cert\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.492520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.492361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-token\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.492520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.492440 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-socket\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.492520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.492461 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-credential-socket\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.492687 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.492546 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-data\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.492687 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.492592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcss\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-kube-api-access-bjcss\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.492687 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.492622 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-certs\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.492687 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.492646 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-podinfo\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.492836 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.492691 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-envoy\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594135 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcss\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-kube-api-access-bjcss\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594135 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594137 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-certs\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594439 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594156 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-podinfo\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594439 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594174 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-envoy\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594439 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594203 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97892a82-4363-435f-b8ac-ea2f866a25ec-istiod-ca-cert\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594439 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594223 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-token\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594439 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-socket\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594684 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594465 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-credential-socket\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594684 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-data\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594684 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594621 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-certs\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594831 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594798 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-socket\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594886 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594855 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-data\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594907 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-credential-socket\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.594996 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.594981 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97892a82-4363-435f-b8ac-ea2f866a25ec-istiod-ca-cert\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.596974 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.596949 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-envoy\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.597077 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.597050 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-podinfo\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.601893 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.601871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-token\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.601985 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.601945 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcss\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-kube-api-access-bjcss\") pod \"e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.710364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.710245 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:42.731039 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.731011 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:42.912350 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.912294 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7"] Jul 08 20:32:42.919063 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:32:42.919038 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97892a82_4363_435f_b8ac_ea2f866a25ec.slice/crio-9a6e5178994faba53223a9f5b433d72c1c46be36279fdc8646a5de7dc054fb03 WatchSource:0}: Error finding container 9a6e5178994faba53223a9f5b433d72c1c46be36279fdc8646a5de7dc054fb03: Status 404 returned error can't find the container with id 9a6e5178994faba53223a9f5b433d72c1c46be36279fdc8646a5de7dc054fb03 Jul 08 20:32:42.921202 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.921161 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:42.921295 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.921251 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:42.921360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:42.921291 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:43.173566 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:43.173525 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" event={"ID":"97892a82-4363-435f-b8ac-ea2f866a25ec","Type":"ContainerStarted","Data":"e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45"} Jul 08 20:32:43.173741 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:43.173590 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" event={"ID":"97892a82-4363-435f-b8ac-ea2f866a25ec","Type":"ContainerStarted","Data":"9a6e5178994faba53223a9f5b433d72c1c46be36279fdc8646a5de7dc054fb03"} Jul 08 20:32:43.193826 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:43.193780 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" podStartSLOduration=1.193764329 podStartE2EDuration="1.193764329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:32:43.191585036 +0000 UTC m=+1931.188601982" watchObservedRunningTime="2026-07-08 20:32:43.193764329 +0000 UTC m=+1931.190781264" Jul 08 20:32:43.731779 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:43.731747 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:43.736744 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:43.736718 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:44.178937 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:44.178907 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:44.179813 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:44.179799 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:32:45.146519 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.146485 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:32:45.508573 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.508545 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:45.582149 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.582119 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:45.584072 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.584051 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts"] Jul 08 20:32:45.587342 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.587298 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:45.590040 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.590021 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-879445-serving-cert\"" Jul 08 20:32:45.600493 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.600471 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts"] Jul 08 20:32:45.660339 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.658012 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:45.726004 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.725979 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd3f198-a481-4f1f-bd00-0b9b861244ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:45.726151 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.726033 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tscm5\" (UniqueName: \"kubernetes.io/projected/bdd3f198-a481-4f1f-bd00-0b9b861244ab-kube-api-access-tscm5\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:45.726214 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.726165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3f198-a481-4f1f-bd00-0b9b861244ab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:45.827462 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.827376 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3f198-a481-4f1f-bd00-0b9b861244ab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:45.827462 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.827434 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd3f198-a481-4f1f-bd00-0b9b861244ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:45.827670 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.827511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tscm5\" (UniqueName: \"kubernetes.io/projected/bdd3f198-a481-4f1f-bd00-0b9b861244ab-kube-api-access-tscm5\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:45.827670 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:32:45.827594 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-serving-cert: secret "maas-api-e2e-inf-b-879445-serving-cert" not found Jul 08 20:32:45.827670 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:32:45.827662 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdd3f198-a481-4f1f-bd00-0b9b861244ab-maas-api-tls podName:bdd3f198-a481-4f1f-bd00-0b9b861244ab nodeName:}" failed. No retries permitted until 2026-07-08 20:32:46.327636368 +0000 UTC m=+1934.324653281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bdd3f198-a481-4f1f-bd00-0b9b861244ab-maas-api-tls") pod "maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" (UID: "bdd3f198-a481-4f1f-bd00-0b9b861244ab") : secret "maas-api-e2e-inf-b-879445-serving-cert" not found Jul 08 20:32:45.828022 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.828005 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3f198-a481-4f1f-bd00-0b9b861244ab-openshift-service-ca\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:45.836708 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:45.836688 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tscm5\" (UniqueName: \"kubernetes.io/projected/bdd3f198-a481-4f1f-bd00-0b9b861244ab-kube-api-access-tscm5\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:46.172450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:46.172416 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:46.271045 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:46.271013 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:46.332880 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:46.332844 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd3f198-a481-4f1f-bd00-0b9b861244ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:46.335217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:46.335198 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd3f198-a481-4f1f-bd00-0b9b861244ab-maas-api-tls\") pod \"maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:46.498209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:46.498182 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:46.633827 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:46.633800 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts"] Jul 08 20:32:46.635563 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:32:46.635533 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd3f198_a481_4f1f_bd00_0b9b861244ab.slice/crio-87b8ca419d2293f56a4f038d63d0400b75beed283cd860afc0adf0797d57da77 WatchSource:0}: Error finding container 87b8ca419d2293f56a4f038d63d0400b75beed283cd860afc0adf0797d57da77: Status 404 returned error can't find the container with id 87b8ca419d2293f56a4f038d63d0400b75beed283cd860afc0adf0797d57da77 Jul 08 20:32:47.196631 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:47.196595 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" event={"ID":"bdd3f198-a481-4f1f-bd00-0b9b861244ab","Type":"ContainerStarted","Data":"97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1"} Jul 08 20:32:47.197132 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:47.196658 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" event={"ID":"bdd3f198-a481-4f1f-bd00-0b9b861244ab","Type":"ContainerStarted","Data":"87b8ca419d2293f56a4f038d63d0400b75beed283cd860afc0adf0797d57da77"} Jul 08 20:32:47.197132 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:47.196813 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:47.214939 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:47.214883 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" podStartSLOduration=1.876025657 podStartE2EDuration="2.214864495s" totalImagesPullingTime="338.838838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:32:47.212925372 +0000 UTC m=+1935.209942295" watchObservedRunningTime="2026-07-08 20:32:47.214864495 +0000 UTC m=+1935.211881429" Jul 08 20:32:49.992722 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:49.992683 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:50.211930 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:50.211902 2573 generic.go:356] "Generic (PLEG): container finished" podID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerID="f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0" exitCode=0 Jul 08 20:32:50.212098 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:50.211972 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" event={"ID":"205ac82a-139d-48c4-bc2a-bb8f1611ace8","Type":"ContainerDied","Data":"f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0"} Jul 08 20:32:50.212405 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:50.212391 2573 scope.go:117] "RemoveContainer" containerID="f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0" Jul 08 20:32:50.213411 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:50.213385 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:50.213510 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:50.213457 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:50.213510 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:50.213503 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:51.111963 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:51.111927 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:51.219562 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:51.219529 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" event={"ID":"205ac82a-139d-48c4-bc2a-bb8f1611ace8","Type":"ContainerStarted","Data":"372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82"} Jul 08 20:32:51.769021 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:51.768984 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:51.769241 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:51.769150 2573 patch_prober.go:28] interesting pod/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 08 20:32:51.769241 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:51.769214 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 08 20:32:52.768050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:52.768014 2573 patch_prober.go:28] interesting pod/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 08 20:32:52.768677 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:52.768645 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 08 20:32:53.207459 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:53.207429 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:32:53.767661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:53.767238 2573 patch_prober.go:28] interesting pod/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 08 20:32:53.767661 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:53.767294 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 08 20:32:54.767785 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:54.767747 2573 patch_prober.go:28] interesting pod/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 08 20:32:54.768416 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:54.768380 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 08 20:32:55.770978 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:55.770898 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:55.771427 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:55.771264 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:55.771963 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:55.771944 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:32:58.686565 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:58.686535 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:58.783620 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:58.783593 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:32:59.180410 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:59.180384 2573 patch_prober.go:28] interesting pod/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 08 20:32:59.180642 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:59.180432 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 08 20:32:59.261681 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:59.261655 2573 generic.go:356] "Generic (PLEG): container finished" podID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerID="e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45" exitCode=0 Jul 08 20:32:59.261810 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:59.261727 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" event={"ID":"97892a82-4363-435f-b8ac-ea2f866a25ec","Type":"ContainerDied","Data":"e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45"} Jul 08 20:32:59.262103 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:59.262088 2573 scope.go:117] "RemoveContainer" containerID="e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45" Jul 08 20:32:59.263082 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:59.263057 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:59.263151 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:59.263135 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:32:59.263201 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:32:59.263185 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:33:00.272121 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:00.272081 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" event={"ID":"97892a82-4363-435f-b8ac-ea2f866a25ec","Type":"ContainerStarted","Data":"205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca"} Jul 08 20:33:00.731740 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:00.731701 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:33:00.731952 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:00.731928 2573 patch_prober.go:28] interesting pod/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 08 20:33:00.732005 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:00.731988 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 08 20:33:01.731689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:01.731656 2573 patch_prober.go:28] interesting pod/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 08 20:33:01.732067 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:01.731715 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 08 20:33:02.732214 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:02.732181 2573 patch_prober.go:28] interesting pod/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 08 20:33:02.732619 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:02.732238 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 08 20:33:03.731407 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:03.731376 2573 patch_prober.go:28] interesting pod/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 08 20:33:03.731580 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:03.731424 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 08 20:33:04.735023 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:04.734991 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:33:04.735407 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:04.735331 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:33:04.735990 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:04.735969 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:33:07.809478 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:07.809443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:33:07.926640 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:07.926617 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:33:11.654990 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:11.654961 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:33:25.661586 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:25.661509 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:33:45.299770 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:33:45.299741 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:34:25.105613 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:25.105580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:34:25.976614 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:25.976586 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:34:26.171511 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:26.171465 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:34:26.677827 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:26.677789 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:34:26.851952 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:26.851903 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:34:26.946905 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:26.946830 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:34:27.097065 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.097032 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t"] Jul 08 20:34:27.097377 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.097350 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" podUID="e2e54471-5158-48a2-b643-0d0c7153ba03" containerName="maas-api" containerID="cri-o://0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a" gracePeriod=30 Jul 08 20:34:27.139037 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.139014 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:34:27.543372 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.543346 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:34:27.644365 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.644294 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e54471-5158-48a2-b643-0d0c7153ba03-openshift-service-ca\") pod \"e2e54471-5158-48a2-b643-0d0c7153ba03\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " Jul 08 20:34:27.644500 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.644396 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e54471-5158-48a2-b643-0d0c7153ba03-maas-api-tls\") pod \"e2e54471-5158-48a2-b643-0d0c7153ba03\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " Jul 08 20:34:27.644500 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.644420 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7w87\" (UniqueName: \"kubernetes.io/projected/e2e54471-5158-48a2-b643-0d0c7153ba03-kube-api-access-f7w87\") pod \"e2e54471-5158-48a2-b643-0d0c7153ba03\" (UID: \"e2e54471-5158-48a2-b643-0d0c7153ba03\") " Jul 08 20:34:27.644621 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.644599 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e54471-5158-48a2-b643-0d0c7153ba03-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2e54471-5158-48a2-b643-0d0c7153ba03" (UID: "e2e54471-5158-48a2-b643-0d0c7153ba03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:34:27.644699 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.644686 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e54471-5158-48a2-b643-0d0c7153ba03-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:34:27.646319 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.646280 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e54471-5158-48a2-b643-0d0c7153ba03-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2e54471-5158-48a2-b643-0d0c7153ba03" (UID: "e2e54471-5158-48a2-b643-0d0c7153ba03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:34:27.646531 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.646512 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e54471-5158-48a2-b643-0d0c7153ba03-kube-api-access-f7w87" (OuterVolumeSpecName: "kube-api-access-f7w87") pod "e2e54471-5158-48a2-b643-0d0c7153ba03" (UID: "e2e54471-5158-48a2-b643-0d0c7153ba03"). InnerVolumeSpecName "kube-api-access-f7w87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:34:27.745892 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.745874 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e54471-5158-48a2-b643-0d0c7153ba03-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:34:27.745892 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.745892 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7w87\" (UniqueName: \"kubernetes.io/projected/e2e54471-5158-48a2-b643-0d0c7153ba03-kube-api-access-f7w87\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:34:27.754361 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.754339 2573 generic.go:356] "Generic (PLEG): container finished" podID="e2e54471-5158-48a2-b643-0d0c7153ba03" containerID="0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a" exitCode=0 Jul 08 20:34:27.754450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.754418 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" event={"ID":"e2e54471-5158-48a2-b643-0d0c7153ba03","Type":"ContainerDied","Data":"0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a"} Jul 08 20:34:27.754450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.754423 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" Jul 08 20:34:27.754525 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.754458 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t" event={"ID":"e2e54471-5158-48a2-b643-0d0c7153ba03","Type":"ContainerDied","Data":"8bd81cd4c6809f942c7858edae8b9cce886b7f623234cf1d95be2bd1fc2e0d09"} Jul 08 20:34:27.754525 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.754483 2573 scope.go:117] "RemoveContainer" containerID="0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a" Jul 08 20:34:27.763051 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.763036 2573 scope.go:117] "RemoveContainer" containerID="0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a" Jul 08 20:34:27.763283 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:34:27.763261 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a\": container with ID starting with 0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a not found: ID does not exist" containerID="0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a" Jul 08 20:34:27.763392 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.763291 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a"} err="failed to get container status \"0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a\": rpc error: code = NotFound desc = could not find container \"0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a\": container with ID starting with 0ba12a3e73d49a224ae38917dc98a5b3fdd00f04d86e23b8acd90f0c47192f8a not found: ID does not exist" Jul 08 20:34:27.778224 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.778203 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t"] Jul 08 20:34:27.782721 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:27.782700 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-930853-f7fff676f-d5c7t"] Jul 08 20:34:28.787165 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:28.787127 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e54471-5158-48a2-b643-0d0c7153ba03" path="/var/lib/kubelet/pods/e2e54471-5158-48a2-b643-0d0c7153ba03/volumes" Jul 08 20:34:31.748125 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:34:31.748086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:08.642464 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:08.642430 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:08.912814 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:08.912728 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx"] Jul 08 20:35:08.913005 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:08.912938 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" containerID="cri-o://372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82" gracePeriod=30 Jul 08 20:35:08.929298 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:08.929269 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:09.362770 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:09.362737 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:09.793443 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:09.793416 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:09.989239 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:09.989187 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:10.091004 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.090924 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:10.233764 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.233730 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts"] Jul 08 20:35:10.233996 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.233928 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" podUID="bdd3f198-a481-4f1f-bd00-0b9b861244ab" containerName="maas-api" containerID="cri-o://97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1" gracePeriod=30 Jul 08 20:35:10.267761 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.267727 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:10.677106 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.677080 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:35:10.741963 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.741933 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd3f198-a481-4f1f-bd00-0b9b861244ab-maas-api-tls\") pod \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " Jul 08 20:35:10.742137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.742025 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tscm5\" (UniqueName: \"kubernetes.io/projected/bdd3f198-a481-4f1f-bd00-0b9b861244ab-kube-api-access-tscm5\") pod \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " Jul 08 20:35:10.742137 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.742100 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3f198-a481-4f1f-bd00-0b9b861244ab-openshift-service-ca\") pod \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\" (UID: \"bdd3f198-a481-4f1f-bd00-0b9b861244ab\") " Jul 08 20:35:10.742487 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.742459 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd3f198-a481-4f1f-bd00-0b9b861244ab-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bdd3f198-a481-4f1f-bd00-0b9b861244ab" (UID: "bdd3f198-a481-4f1f-bd00-0b9b861244ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:35:10.744181 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.744152 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd3f198-a481-4f1f-bd00-0b9b861244ab-kube-api-access-tscm5" (OuterVolumeSpecName: "kube-api-access-tscm5") pod "bdd3f198-a481-4f1f-bd00-0b9b861244ab" (UID: "bdd3f198-a481-4f1f-bd00-0b9b861244ab"). InnerVolumeSpecName "kube-api-access-tscm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:35:10.744265 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.744201 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdd3f198-a481-4f1f-bd00-0b9b861244ab-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bdd3f198-a481-4f1f-bd00-0b9b861244ab" (UID: "bdd3f198-a481-4f1f-bd00-0b9b861244ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:35:10.773727 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.773665 2573 patch_prober.go:28] interesting pod/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 20:35:10.773821 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.773726 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 20:35:10.843593 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.843560 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdd3f198-a481-4f1f-bd00-0b9b861244ab-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:10.843593 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.843587 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdd3f198-a481-4f1f-bd00-0b9b861244ab-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:10.843593 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.843596 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tscm5\" (UniqueName: \"kubernetes.io/projected/bdd3f198-a481-4f1f-bd00-0b9b861244ab-kube-api-access-tscm5\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:10.987245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.987207 2573 generic.go:356] "Generic (PLEG): container finished" podID="bdd3f198-a481-4f1f-bd00-0b9b861244ab" containerID="97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1" exitCode=0 Jul 08 20:35:10.987473 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.987254 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" Jul 08 20:35:10.987473 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.987302 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" event={"ID":"bdd3f198-a481-4f1f-bd00-0b9b861244ab","Type":"ContainerDied","Data":"97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1"} Jul 08 20:35:10.987473 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.987389 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts" event={"ID":"bdd3f198-a481-4f1f-bd00-0b9b861244ab","Type":"ContainerDied","Data":"87b8ca419d2293f56a4f038d63d0400b75beed283cd860afc0adf0797d57da77"} Jul 08 20:35:10.987473 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.987418 2573 scope.go:117] "RemoveContainer" containerID="97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1" Jul 08 20:35:10.998051 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.998026 2573 scope.go:117] "RemoveContainer" containerID="97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1" Jul 08 20:35:10.998394 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:10.998377 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1\": container with ID starting with 97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1 not found: ID does not exist" containerID="97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1" Jul 08 20:35:10.998470 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:10.998400 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1"} err="failed to get container status \"97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1\": rpc error: code = NotFound desc = could not find container \"97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1\": container with ID starting with 97d024ee1958286e83c92eae3590297ea939f594b287bcb5b3473c8778a0bcb1 not found: ID does not exist" Jul 08 20:35:11.004245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:11.004222 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts"] Jul 08 20:35:11.007211 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:11.007192 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-879445-5c96f58f69-dp5ts"] Jul 08 20:35:12.496924 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:12.496881 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:12.788587 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:12.788505 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd3f198-a481-4f1f-bd00-0b9b861244ab" path="/var/lib/kubelet/pods/bdd3f198-a481-4f1f-bd00-0b9b861244ab/volumes" Jul 08 20:35:14.165781 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.165759 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:35:14.173740 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.173696 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-data\") pod \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " Jul 08 20:35:14.173740 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.173720 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-credential-socket\") pod \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " Jul 08 20:35:14.173833 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.173741 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-podinfo\") pod \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " Jul 08 20:35:14.173833 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.173763 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-envoy\") pod \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " Jul 08 20:35:14.173833 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.173797 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istiod-ca-cert\") pod \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " Jul 08 20:35:14.173940 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.173873 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gstcd\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-kube-api-access-gstcd\") pod \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " Jul 08 20:35:14.173940 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.173920 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-token\") pod \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " Jul 08 20:35:14.174034 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.173972 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "205ac82a-139d-48c4-bc2a-bb8f1611ace8" (UID: "205ac82a-139d-48c4-bc2a-bb8f1611ace8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:14.174034 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.173978 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-socket\") pod \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " Jul 08 20:35:14.174034 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.174029 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-certs\") pod \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\" (UID: \"205ac82a-139d-48c4-bc2a-bb8f1611ace8\") " Jul 08 20:35:14.174190 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.174168 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "205ac82a-139d-48c4-bc2a-bb8f1611ace8" (UID: "205ac82a-139d-48c4-bc2a-bb8f1611ace8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:14.174264 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.174246 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "205ac82a-139d-48c4-bc2a-bb8f1611ace8" (UID: "205ac82a-139d-48c4-bc2a-bb8f1611ace8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:35:14.174460 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.174439 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "205ac82a-139d-48c4-bc2a-bb8f1611ace8" (UID: "205ac82a-139d-48c4-bc2a-bb8f1611ace8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:14.174548 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.174528 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-data" (OuterVolumeSpecName: "istio-data") pod "205ac82a-139d-48c4-bc2a-bb8f1611ace8" (UID: "205ac82a-139d-48c4-bc2a-bb8f1611ace8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:14.174617 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.174584 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:14.174676 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.174617 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:14.174676 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.174630 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:14.174676 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.174645 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:14.175868 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.175842 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "205ac82a-139d-48c4-bc2a-bb8f1611ace8" (UID: "205ac82a-139d-48c4-bc2a-bb8f1611ace8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:14.176008 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.175992 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-kube-api-access-gstcd" (OuterVolumeSpecName: "kube-api-access-gstcd") pod "205ac82a-139d-48c4-bc2a-bb8f1611ace8" (UID: "205ac82a-139d-48c4-bc2a-bb8f1611ace8"). InnerVolumeSpecName "kube-api-access-gstcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:35:14.176072 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.176056 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-token" (OuterVolumeSpecName: "istio-token") pod "205ac82a-139d-48c4-bc2a-bb8f1611ace8" (UID: "205ac82a-139d-48c4-bc2a-bb8f1611ace8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:35:14.176205 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.176188 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "205ac82a-139d-48c4-bc2a-bb8f1611ace8" (UID: "205ac82a-139d-48c4-bc2a-bb8f1611ace8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:35:14.275262 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.275238 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:14.275262 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.275259 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:14.275426 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.275268 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:14.275426 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.275277 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/205ac82a-139d-48c4-bc2a-bb8f1611ace8-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:14.275426 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.275285 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstcd\" (UniqueName: \"kubernetes.io/projected/205ac82a-139d-48c4-bc2a-bb8f1611ace8-kube-api-access-gstcd\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:14.468194 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.468138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:14.563590 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:14.563566 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:15.010559 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.010535 2573 generic.go:356] "Generic (PLEG): container finished" podID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerID="372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82" exitCode=0 Jul 08 20:35:15.010688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.010582 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" Jul 08 20:35:15.010688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.010600 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" event={"ID":"205ac82a-139d-48c4-bc2a-bb8f1611ace8","Type":"ContainerDied","Data":"372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82"} Jul 08 20:35:15.010688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.010651 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx" event={"ID":"205ac82a-139d-48c4-bc2a-bb8f1611ace8","Type":"ContainerDied","Data":"e94bbc6df476e52c57d160d37f61e542f8b0049ef3fdd0e8ffb598ef9b5b327b"} Jul 08 20:35:15.010688 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.010682 2573 scope.go:117] "RemoveContainer" containerID="372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82" Jul 08 20:35:15.019716 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.019699 2573 scope.go:117] "RemoveContainer" containerID="f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0" Jul 08 20:35:15.027414 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.027396 2573 scope.go:117] "RemoveContainer" containerID="372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82" Jul 08 20:35:15.027669 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:15.027650 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82\": container with ID starting with 372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82 not found: ID does not exist" containerID="372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82" Jul 08 20:35:15.027712 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.027677 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82"} err="failed to get container status \"372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82\": rpc error: code = NotFound desc = could not find container \"372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82\": container with ID starting with 372779feb45151ccaa1ac3aad1df958062606f0f0d0a8c58ae561d5169a80b82 not found: ID does not exist" Jul 08 20:35:15.027712 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.027692 2573 scope.go:117] "RemoveContainer" containerID="f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0" Jul 08 20:35:15.027929 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:15.027912 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0\": container with ID starting with f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0 not found: ID does not exist" containerID="f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0" Jul 08 20:35:15.027969 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.027933 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0"} err="failed to get container status \"f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0\": rpc error: code = NotFound desc = could not find container \"f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0\": container with ID starting with f1dd03d8f07c52f9cef673198f22d1399fba63986c68c5bbf73c6d846f3758e0 not found: ID does not exist" Jul 08 20:35:15.033469 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.033449 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx"] Jul 08 20:35:15.037467 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:15.037437 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-930853-openshift-default-85fb557dc7-lmdxx"] Jul 08 20:35:16.313985 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:16.313946 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:16.492570 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:16.492536 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:16.665035 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:16.664969 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7"] Jul 08 20:35:16.665210 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:16.665185 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" containerID="cri-o://205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca" gracePeriod=30 Jul 08 20:35:16.704525 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:16.704498 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:16.786888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:16.786863 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" path="/var/lib/kubelet/pods/205ac82a-139d-48c4-bc2a-bb8f1611ace8/volumes" Jul 08 20:35:17.269818 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.269787 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:17.572438 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.572362 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf"] Jul 08 20:35:17.573012 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.572991 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" Jul 08 20:35:17.573012 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.573013 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" Jul 08 20:35:17.573156 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.573033 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd3f198-a481-4f1f-bd00-0b9b861244ab" containerName="maas-api" Jul 08 20:35:17.573156 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.573040 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd3f198-a481-4f1f-bd00-0b9b861244ab" containerName="maas-api" Jul 08 20:35:17.573156 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.573077 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e54471-5158-48a2-b643-0d0c7153ba03" containerName="maas-api" Jul 08 20:35:17.573156 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.573085 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e54471-5158-48a2-b643-0d0c7153ba03" containerName="maas-api" Jul 08 20:35:17.573375 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.573189 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" Jul 08 20:35:17.573375 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.573200 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd3f198-a481-4f1f-bd00-0b9b861244ab" containerName="maas-api" Jul 08 20:35:17.573375 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.573211 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e54471-5158-48a2-b643-0d0c7153ba03" containerName="maas-api" Jul 08 20:35:17.573375 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.573220 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" Jul 08 20:35:17.577713 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.577694 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.580388 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.580369 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-575204-openshift-default-dockercfg-76jqw\"" Jul 08 20:35:17.589038 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.589004 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf"] Jul 08 20:35:17.595754 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.595729 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:17.604667 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.604634 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-certs\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.604785 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.604712 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7128e6-0c54-4727-87cc-39be90e600e4-istiod-ca-cert\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.604785 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.604739 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-envoy\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.604894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.604787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-token\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.604894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.604814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-data\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.604894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.604873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-socket\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.605058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.604911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-credential-socket\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.605058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.604932 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fzf8\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-kube-api-access-5fzf8\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.605058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.604970 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-podinfo\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.705988 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.705961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-token\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706109 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706002 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-data\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706109 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-socket\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706185 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706108 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-credential-socket\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706185 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fzf8\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-kube-api-access-5fzf8\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706185 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-podinfo\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706351 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706211 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-certs\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706351 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706250 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7128e6-0c54-4727-87cc-39be90e600e4-istiod-ca-cert\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706351 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-envoy\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706827 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706494 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-data\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706827 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706540 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-socket\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706827 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706572 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-credential-socket\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.706827 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.706608 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-certs\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.707151 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.707129 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7128e6-0c54-4727-87cc-39be90e600e4-istiod-ca-cert\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.708414 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.708394 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-envoy\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.708537 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.708520 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-podinfo\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.714483 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.714464 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fzf8\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-kube-api-access-5fzf8\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.714565 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.714526 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-token\") pod \"e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.890751 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.890678 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:17.965067 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:17.965031 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:18.026642 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:18.026621 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf"] Jul 08 20:35:18.028817 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:35:18.028788 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7128e6_0c54_4727_87cc_39be90e600e4.slice/crio-4729013454cf63546fce999fdea546067c4f68427ebe6bb8038a3963c772d534 WatchSource:0}: Error finding container 4729013454cf63546fce999fdea546067c4f68427ebe6bb8038a3963c772d534: Status 404 returned error can't find the container with id 4729013454cf63546fce999fdea546067c4f68427ebe6bb8038a3963c772d534 Jul 08 20:35:18.030818 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:18.030787 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:18.030928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:18.030869 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:18.030928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:18.030916 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:19.034083 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:19.034047 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" event={"ID":"8e7128e6-0c54-4727-87cc-39be90e600e4","Type":"ContainerStarted","Data":"97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b"} Jul 08 20:35:19.034487 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:19.034105 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" event={"ID":"8e7128e6-0c54-4727-87cc-39be90e600e4","Type":"ContainerStarted","Data":"4729013454cf63546fce999fdea546067c4f68427ebe6bb8038a3963c772d534"} Jul 08 20:35:19.056407 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:19.056362 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" podStartSLOduration=2.056347656 podStartE2EDuration="2.056347656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:35:19.054531497 +0000 UTC m=+2087.051548433" watchObservedRunningTime="2026-07-08 20:35:19.056347656 +0000 UTC m=+2087.053364591" Jul 08 20:35:19.737380 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:19.737346 2573 patch_prober.go:28] interesting pod/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 20:35:19.737550 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:19.737410 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 20:35:19.890959 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:19.890929 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:19.895425 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:19.895402 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:20.039665 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:20.039605 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:20.040477 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:20.040452 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:21.149347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.149299 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:21.205638 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.205611 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g"] Jul 08 20:35:21.206050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.206036 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" Jul 08 20:35:21.206120 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.206051 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="205ac82a-139d-48c4-bc2a-bb8f1611ace8" containerName="istio-proxy" Jul 08 20:35:21.209247 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.209224 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.212151 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.212129 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-575204-serving-cert\"" Jul 08 20:35:21.212266 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.212149 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x22jf\"" Jul 08 20:35:21.212266 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.212131 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:35:21.213256 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.213241 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:35:21.220266 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.220248 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g"] Jul 08 20:35:21.236168 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.236145 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccea67-8ce3-46db-a616-860433e01394-openshift-service-ca\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.236249 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.236183 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bccea67-8ce3-46db-a616-860433e01394-maas-api-tls\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.236293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.236250 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwvl\" (UniqueName: \"kubernetes.io/projected/9bccea67-8ce3-46db-a616-860433e01394-kube-api-access-plwvl\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.251808 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.251786 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:21.316883 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.316855 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:21.337025 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.337000 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwvl\" (UniqueName: \"kubernetes.io/projected/9bccea67-8ce3-46db-a616-860433e01394-kube-api-access-plwvl\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.337119 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.337064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccea67-8ce3-46db-a616-860433e01394-openshift-service-ca\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.337119 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.337109 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bccea67-8ce3-46db-a616-860433e01394-maas-api-tls\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.337249 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:21.337234 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-575204-serving-cert: secret "maas-api-e2e-api-a-575204-serving-cert" not found Jul 08 20:35:21.337329 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:21.337299 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9bccea67-8ce3-46db-a616-860433e01394-maas-api-tls podName:9bccea67-8ce3-46db-a616-860433e01394 nodeName:}" failed. No retries permitted until 2026-07-08 20:35:21.837283975 +0000 UTC m=+2089.834300889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9bccea67-8ce3-46db-a616-860433e01394-maas-api-tls") pod "maas-api-e2e-api-a-575204-577759846c-m9q5g" (UID: "9bccea67-8ce3-46db-a616-860433e01394") : secret "maas-api-e2e-api-a-575204-serving-cert" not found Jul 08 20:35:21.337611 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.337594 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccea67-8ce3-46db-a616-860433e01394-openshift-service-ca\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.349766 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.349746 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwvl\" (UniqueName: \"kubernetes.io/projected/9bccea67-8ce3-46db-a616-860433e01394-kube-api-access-plwvl\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.520857 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.520831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:21.740075 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.740053 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:21.840167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.840142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:21.842593 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.842561 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bccea67-8ce3-46db-a616-860433e01394-maas-api-tls\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.845639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.845615 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bccea67-8ce3-46db-a616-860433e01394-maas-api-tls\") pod \"maas-api-e2e-api-a-575204-577759846c-m9q5g\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:21.917138 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.917119 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:35:21.943346 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943319 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjcss\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-kube-api-access-bjcss\") pod \"97892a82-4363-435f-b8ac-ea2f866a25ec\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " Jul 08 20:35:21.943476 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943393 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97892a82-4363-435f-b8ac-ea2f866a25ec-istiod-ca-cert\") pod \"97892a82-4363-435f-b8ac-ea2f866a25ec\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " Jul 08 20:35:21.943476 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943464 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-certs\") pod \"97892a82-4363-435f-b8ac-ea2f866a25ec\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " Jul 08 20:35:21.943573 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943504 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-socket\") pod \"97892a82-4363-435f-b8ac-ea2f866a25ec\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " Jul 08 20:35:21.943573 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943563 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-podinfo\") pod \"97892a82-4363-435f-b8ac-ea2f866a25ec\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " Jul 08 20:35:21.943669 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943589 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-data\") pod \"97892a82-4363-435f-b8ac-ea2f866a25ec\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " Jul 08 20:35:21.943669 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943620 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-token\") pod \"97892a82-4363-435f-b8ac-ea2f866a25ec\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " Jul 08 20:35:21.943767 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943674 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-envoy\") pod \"97892a82-4363-435f-b8ac-ea2f866a25ec\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " Jul 08 20:35:21.943767 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943714 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-credential-socket\") pod \"97892a82-4363-435f-b8ac-ea2f866a25ec\" (UID: \"97892a82-4363-435f-b8ac-ea2f866a25ec\") " Jul 08 20:35:21.943767 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943728 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97892a82-4363-435f-b8ac-ea2f866a25ec" (UID: "97892a82-4363-435f-b8ac-ea2f866a25ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:21.943914 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.943761 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97892a82-4363-435f-b8ac-ea2f866a25ec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97892a82-4363-435f-b8ac-ea2f866a25ec" (UID: "97892a82-4363-435f-b8ac-ea2f866a25ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:35:21.945113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.944073 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97892a82-4363-435f-b8ac-ea2f866a25ec-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:21.945113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.944095 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:21.945113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.944099 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97892a82-4363-435f-b8ac-ea2f866a25ec" (UID: "97892a82-4363-435f-b8ac-ea2f866a25ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:21.945113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.944295 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97892a82-4363-435f-b8ac-ea2f866a25ec" (UID: "97892a82-4363-435f-b8ac-ea2f866a25ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:21.945113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.945043 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-data" (OuterVolumeSpecName: "istio-data") pod "97892a82-4363-435f-b8ac-ea2f866a25ec" (UID: "97892a82-4363-435f-b8ac-ea2f866a25ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:21.945561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.945541 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-kube-api-access-bjcss" (OuterVolumeSpecName: "kube-api-access-bjcss") pod "97892a82-4363-435f-b8ac-ea2f866a25ec" (UID: "97892a82-4363-435f-b8ac-ea2f866a25ec"). InnerVolumeSpecName "kube-api-access-bjcss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:35:21.945933 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.945915 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97892a82-4363-435f-b8ac-ea2f866a25ec" (UID: "97892a82-4363-435f-b8ac-ea2f866a25ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:21.946024 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.945952 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97892a82-4363-435f-b8ac-ea2f866a25ec" (UID: "97892a82-4363-435f-b8ac-ea2f866a25ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:35:21.946275 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:21.946253 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-token" (OuterVolumeSpecName: "istio-token") pod "97892a82-4363-435f-b8ac-ea2f866a25ec" (UID: "97892a82-4363-435f-b8ac-ea2f866a25ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:35:22.045279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.045219 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:22.045279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.045240 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjcss\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-kube-api-access-bjcss\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:22.045279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.045251 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:22.045279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.045259 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:22.045279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.045267 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:22.045279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.045274 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:22.045279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.045281 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97892a82-4363-435f-b8ac-ea2f866a25ec-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:22.050955 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.050929 2573 generic.go:356] "Generic (PLEG): container finished" podID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerID="205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca" exitCode=0 Jul 08 20:35:22.051048 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.050996 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" Jul 08 20:35:22.051048 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.051006 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" event={"ID":"97892a82-4363-435f-b8ac-ea2f866a25ec","Type":"ContainerDied","Data":"205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca"} Jul 08 20:35:22.051116 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.051050 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7" event={"ID":"97892a82-4363-435f-b8ac-ea2f866a25ec","Type":"ContainerDied","Data":"9a6e5178994faba53223a9f5b433d72c1c46be36279fdc8646a5de7dc054fb03"} Jul 08 20:35:22.051116 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.051071 2573 scope.go:117] "RemoveContainer" containerID="205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca" Jul 08 20:35:22.062032 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.062015 2573 scope.go:117] "RemoveContainer" containerID="e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45" Jul 08 20:35:22.069478 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.069464 2573 scope.go:117] "RemoveContainer" containerID="205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca" Jul 08 20:35:22.069748 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:22.069700 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca\": container with ID starting with 205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca not found: ID does not exist" containerID="205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca" Jul 08 20:35:22.069852 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.069747 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca"} err="failed to get container status \"205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca\": rpc error: code = NotFound desc = could not find container \"205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca\": container with ID starting with 205b9d2bd1ddff887d8497fffc2b89ee8705eaf2199168441bab2ff1b659b9ca not found: ID does not exist" Jul 08 20:35:22.069852 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.069769 2573 scope.go:117] "RemoveContainer" containerID="e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45" Jul 08 20:35:22.070072 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:22.070044 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45\": container with ID starting with e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45 not found: ID does not exist" containerID="e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45" Jul 08 20:35:22.070143 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.070074 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45"} err="failed to get container status \"e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45\": rpc error: code = NotFound desc = could not find container \"e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45\": container with ID starting with e8d2df0957fc27a8235849f0d72917def27c5695df2ba91c78dbcc5a70535e45 not found: ID does not exist" Jul 08 20:35:22.076170 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.076151 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7"] Jul 08 20:35:22.083888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.083869 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-879445-openshift-default-7978cd89df-ttjv7"] Jul 08 20:35:22.120841 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.120823 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:22.249066 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.249045 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g"] Jul 08 20:35:22.250773 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:35:22.250753 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bccea67_8ce3_46db_a616_860433e01394.slice/crio-9ac8d76a7de3957775a35540f8322bae0bb28ff072cf55215590ae7f02ddb800 WatchSource:0}: Error finding container 9ac8d76a7de3957775a35540f8322bae0bb28ff072cf55215590ae7f02ddb800: Status 404 returned error can't find the container with id 9ac8d76a7de3957775a35540f8322bae0bb28ff072cf55215590ae7f02ddb800 Jul 08 20:35:22.787234 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:22.787187 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" path="/var/lib/kubelet/pods/97892a82-4363-435f-b8ac-ea2f866a25ec/volumes" Jul 08 20:35:23.058710 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:23.058634 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" event={"ID":"9bccea67-8ce3-46db-a616-860433e01394","Type":"ContainerStarted","Data":"5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605"} Jul 08 20:35:23.058710 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:23.058680 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" event={"ID":"9bccea67-8ce3-46db-a616-860433e01394","Type":"ContainerStarted","Data":"9ac8d76a7de3957775a35540f8322bae0bb28ff072cf55215590ae7f02ddb800"} Jul 08 20:35:23.058890 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:23.058773 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:23.076651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:23.076608 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" podStartSLOduration=1.487983774 podStartE2EDuration="2.076595114s" totalImagesPullingTime="588.61134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:35:23.076106927 +0000 UTC m=+2091.073123862" watchObservedRunningTime="2026-07-08 20:35:23.076595114 +0000 UTC m=+2091.073612048" Jul 08 20:35:25.622177 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:25.622150 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:26.146050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.146019 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:26.491209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.491187 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws"] Jul 08 20:35:26.491772 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.491752 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" Jul 08 20:35:26.491772 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.491775 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" Jul 08 20:35:26.491990 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.491812 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" Jul 08 20:35:26.491990 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.491822 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" Jul 08 20:35:26.491990 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.491929 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" Jul 08 20:35:26.491990 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.491946 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="97892a82-4363-435f-b8ac-ea2f866a25ec" containerName="istio-proxy" Jul 08 20:35:26.495180 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.495165 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.498152 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.498136 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2f232c-openshift-default-dockercfg-4ll6x\"" Jul 08 20:35:26.508950 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.508921 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws"] Jul 08 20:35:26.520524 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.520503 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:26.583984 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.583960 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-certs\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.584083 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.584035 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-data\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.584083 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.584076 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-token\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.584191 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.584102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istiod-ca-cert\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.584191 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.584165 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-credential-socket\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.584260 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.584220 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-socket\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.584260 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.584245 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-envoy\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.584345 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.584271 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-podinfo\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.584345 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.584297 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zsrh\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-kube-api-access-7zsrh\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.656061 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.656034 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:26.685009 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.684984 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-certs\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685097 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-data\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685097 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685078 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-token\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685217 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685105 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istiod-ca-cert\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685270 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685234 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-credential-socket\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685349 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-socket\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685349 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-envoy\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685370 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-podinfo\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685396 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-certs\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685450 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685400 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zsrh\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-kube-api-access-7zsrh\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685610 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685588 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-credential-socket\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685693 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685665 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-socket\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685693 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685686 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-data\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.685881 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.685864 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istiod-ca-cert\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.687527 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.687508 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-envoy\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.687683 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.687662 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-podinfo\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.693625 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.693605 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-token\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.693814 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.693794 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zsrh\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-kube-api-access-7zsrh\") pod \"e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.805365 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.805294 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:26.934150 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.934124 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws"] Jul 08 20:35:26.937129 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:35:26.937100 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba0da2c_9080_4a7e_b955_e1e2640a45b7.slice/crio-87cc575e599f7c36d0f90f4c62b47181c07441f6ff5b6e967b6cc1ae8bcc09de WatchSource:0}: Error finding container 87cc575e599f7c36d0f90f4c62b47181c07441f6ff5b6e967b6cc1ae8bcc09de: Status 404 returned error can't find the container with id 87cc575e599f7c36d0f90f4c62b47181c07441f6ff5b6e967b6cc1ae8bcc09de Jul 08 20:35:26.939421 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.939391 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:26.939503 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.939454 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:26.939503 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:26.939484 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:27.082992 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:27.082903 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" event={"ID":"5ba0da2c-9080-4a7e-b955-e1e2640a45b7","Type":"ContainerStarted","Data":"2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0"} Jul 08 20:35:27.082992 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:27.082961 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" event={"ID":"5ba0da2c-9080-4a7e-b955-e1e2640a45b7","Type":"ContainerStarted","Data":"87cc575e599f7c36d0f90f4c62b47181c07441f6ff5b6e967b6cc1ae8bcc09de"} Jul 08 20:35:27.108078 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:27.108015 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" podStartSLOduration=1.1079954459999999 podStartE2EDuration="1.107995446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:35:27.104251664 +0000 UTC m=+2095.101268599" watchObservedRunningTime="2026-07-08 20:35:27.107995446 +0000 UTC m=+2095.105012382" Jul 08 20:35:27.805692 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:27.805663 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:27.810562 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:27.810540 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:28.089355 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:28.089261 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:28.090044 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:28.090026 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:29.069406 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.069380 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:29.077724 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.077697 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:29.156556 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.156519 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq"] Jul 08 20:35:29.180479 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.179996 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:29.185470 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.182241 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.185639 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.185617 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2f232c-serving-cert\"" Jul 08 20:35:29.190983 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.190955 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq"] Jul 08 20:35:29.261185 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.261142 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:29.309904 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.309870 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a538285-e988-4d96-937c-541741bd3c65-maas-api-tls\") pod \"maas-api-e2e-api-b-2f232c-f47c96779-sb8pq\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.310079 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.309927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a538285-e988-4d96-937c-541741bd3c65-openshift-service-ca\") pod \"maas-api-e2e-api-b-2f232c-f47c96779-sb8pq\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.310079 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.309963 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmd8\" (UniqueName: \"kubernetes.io/projected/0a538285-e988-4d96-937c-541741bd3c65-kube-api-access-dhmd8\") pod \"maas-api-e2e-api-b-2f232c-f47c96779-sb8pq\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.359192 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.359096 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:29.411222 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.411172 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmd8\" (UniqueName: \"kubernetes.io/projected/0a538285-e988-4d96-937c-541741bd3c65-kube-api-access-dhmd8\") pod \"maas-api-e2e-api-b-2f232c-f47c96779-sb8pq\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.411429 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.411339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a538285-e988-4d96-937c-541741bd3c65-maas-api-tls\") pod \"maas-api-e2e-api-b-2f232c-f47c96779-sb8pq\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.411429 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.411392 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a538285-e988-4d96-937c-541741bd3c65-openshift-service-ca\") pod \"maas-api-e2e-api-b-2f232c-f47c96779-sb8pq\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.412089 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.412072 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a538285-e988-4d96-937c-541741bd3c65-openshift-service-ca\") pod \"maas-api-e2e-api-b-2f232c-f47c96779-sb8pq\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.413839 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.413819 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a538285-e988-4d96-937c-541741bd3c65-maas-api-tls\") pod \"maas-api-e2e-api-b-2f232c-f47c96779-sb8pq\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.419745 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.419720 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmd8\" (UniqueName: \"kubernetes.io/projected/0a538285-e988-4d96-937c-541741bd3c65-kube-api-access-dhmd8\") pod \"maas-api-e2e-api-b-2f232c-f47c96779-sb8pq\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.499098 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.499065 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:29.650157 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.650072 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:29.692821 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.692797 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq"] Jul 08 20:35:29.694931 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:35:29.694903 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a538285_e988_4d96_937c_541741bd3c65.slice/crio-efc5a6fcfa8f84116e2b2a9eae8e314b27fff3149ba96e1c4b2927460a43123c WatchSource:0}: Error finding container efc5a6fcfa8f84116e2b2a9eae8e314b27fff3149ba96e1c4b2927460a43123c: Status 404 returned error can't find the container with id efc5a6fcfa8f84116e2b2a9eae8e314b27fff3149ba96e1c4b2927460a43123c Jul 08 20:35:29.763332 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:29.762456 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:30.106374 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:30.106339 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" event={"ID":"0a538285-e988-4d96-937c-541741bd3c65","Type":"ContainerStarted","Data":"efc5a6fcfa8f84116e2b2a9eae8e314b27fff3149ba96e1c4b2927460a43123c"} Jul 08 20:35:31.113070 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:31.113037 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" event={"ID":"0a538285-e988-4d96-937c-541741bd3c65","Type":"ContainerStarted","Data":"d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59"} Jul 08 20:35:31.113463 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:31.113094 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:31.132915 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:31.132874 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" podStartSLOduration=1.44145907 podStartE2EDuration="2.132859653s" totalImagesPullingTime="691.400583ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:35:31.131010017 +0000 UTC m=+2099.128026951" watchObservedRunningTime="2026-07-08 20:35:31.132859653 +0000 UTC m=+2099.129876587" Jul 08 20:35:32.711792 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:32.711765 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:35:32.718996 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:32.718978 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:35:33.645255 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:33.645222 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:36.081480 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:36.081446 2573 patch_prober.go:28] interesting pod/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 20:35:36.081896 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:36.081515 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 20:35:37.123739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:37.123714 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:37.152244 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:37.152201 2573 generic.go:356] "Generic (PLEG): container finished" podID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerID="97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b" exitCode=0 Jul 08 20:35:37.152439 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:37.152372 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" event={"ID":"8e7128e6-0c54-4727-87cc-39be90e600e4","Type":"ContainerDied","Data":"97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b"} Jul 08 20:35:37.152815 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:37.152789 2573 scope.go:117] "RemoveContainer" containerID="97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b" Jul 08 20:35:37.154222 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:37.154185 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:37.154339 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:37.154272 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:37.158429 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:37.158395 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:38.165437 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:38.164359 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" event={"ID":"8e7128e6-0c54-4727-87cc-39be90e600e4","Type":"ContainerStarted","Data":"5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8"} Jul 08 20:35:38.891545 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:38.891505 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:38.891802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:38.891635 2573 patch_prober.go:28] interesting pod/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 08 20:35:38.891802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:38.891677 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 08 20:35:39.891708 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:39.891627 2573 patch_prober.go:28] interesting pod/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 08 20:35:39.891708 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:39.891681 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 08 20:35:40.891421 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:40.891383 2573 patch_prober.go:28] interesting pod/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 08 20:35:40.891624 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:40.891437 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 08 20:35:41.891075 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:41.891045 2573 patch_prober.go:28] interesting pod/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 08 20:35:41.891467 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:41.891096 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 08 20:35:42.089675 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.089643 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:42.569666 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.569630 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:42.670416 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.670389 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:42.770205 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.770171 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:42.870111 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.870035 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g"] Jul 08 20:35:42.870287 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.870261 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" podUID="9bccea67-8ce3-46db-a616-860433e01394" containerName="maas-api" containerID="cri-o://5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605" gracePeriod=30 Jul 08 20:35:42.909422 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.901112 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:42.909422 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.901998 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:42.909422 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.902753 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:42.909422 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:42.902926 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:43.090347 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.090298 2573 patch_prober.go:28] interesting pod/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 08 20:35:43.090468 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.090378 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 08 20:35:43.121304 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.121259 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:43.193430 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.193405 2573 generic.go:356] "Generic (PLEG): container finished" podID="9bccea67-8ce3-46db-a616-860433e01394" containerID="5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605" exitCode=0 Jul 08 20:35:43.193559 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.193446 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" Jul 08 20:35:43.193559 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.193495 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" event={"ID":"9bccea67-8ce3-46db-a616-860433e01394","Type":"ContainerDied","Data":"5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605"} Jul 08 20:35:43.193672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.193572 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g" event={"ID":"9bccea67-8ce3-46db-a616-860433e01394","Type":"ContainerDied","Data":"9ac8d76a7de3957775a35540f8322bae0bb28ff072cf55215590ae7f02ddb800"} Jul 08 20:35:43.193672 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.193603 2573 scope.go:117] "RemoveContainer" containerID="5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605" Jul 08 20:35:43.195019 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.194999 2573 generic.go:356] "Generic (PLEG): container finished" podID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerID="2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0" exitCode=0 Jul 08 20:35:43.195117 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.195077 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" event={"ID":"5ba0da2c-9080-4a7e-b955-e1e2640a45b7","Type":"ContainerDied","Data":"2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0"} Jul 08 20:35:43.195549 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.195534 2573 scope.go:117] "RemoveContainer" containerID="2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0" Jul 08 20:35:43.196411 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.196385 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:43.196512 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.196462 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:43.196581 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.196519 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:43.202070 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.202056 2573 scope.go:117] "RemoveContainer" containerID="5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605" Jul 08 20:35:43.202331 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:43.202280 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605\": container with ID starting with 5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605 not found: ID does not exist" containerID="5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605" Jul 08 20:35:43.202331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.202317 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605"} err="failed to get container status \"5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605\": rpc error: code = NotFound desc = could not find container \"5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605\": container with ID starting with 5757c4a075c73e98c827ecf7d2b0623e89790b559920dca0af0cb1f171387605 not found: ID does not exist" Jul 08 20:35:43.265625 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.265604 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plwvl\" (UniqueName: \"kubernetes.io/projected/9bccea67-8ce3-46db-a616-860433e01394-kube-api-access-plwvl\") pod \"9bccea67-8ce3-46db-a616-860433e01394\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " Jul 08 20:35:43.265742 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.265670 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccea67-8ce3-46db-a616-860433e01394-openshift-service-ca\") pod \"9bccea67-8ce3-46db-a616-860433e01394\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " Jul 08 20:35:43.265803 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.265749 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bccea67-8ce3-46db-a616-860433e01394-maas-api-tls\") pod \"9bccea67-8ce3-46db-a616-860433e01394\" (UID: \"9bccea67-8ce3-46db-a616-860433e01394\") " Jul 08 20:35:43.266500 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.266475 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bccea67-8ce3-46db-a616-860433e01394-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9bccea67-8ce3-46db-a616-860433e01394" (UID: "9bccea67-8ce3-46db-a616-860433e01394"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:35:43.268492 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.268468 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bccea67-8ce3-46db-a616-860433e01394-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9bccea67-8ce3-46db-a616-860433e01394" (UID: "9bccea67-8ce3-46db-a616-860433e01394"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:35:43.268492 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.268479 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bccea67-8ce3-46db-a616-860433e01394-kube-api-access-plwvl" (OuterVolumeSpecName: "kube-api-access-plwvl") pod "9bccea67-8ce3-46db-a616-860433e01394" (UID: "9bccea67-8ce3-46db-a616-860433e01394"). InnerVolumeSpecName "kube-api-access-plwvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:35:43.367653 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.367609 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9bccea67-8ce3-46db-a616-860433e01394-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:43.367653 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.367645 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwvl\" (UniqueName: \"kubernetes.io/projected/9bccea67-8ce3-46db-a616-860433e01394-kube-api-access-plwvl\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:43.367653 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.367661 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bccea67-8ce3-46db-a616-860433e01394-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:43.517340 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.517294 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g"] Jul 08 20:35:43.523233 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:43.523214 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-575204-577759846c-m9q5g"] Jul 08 20:35:44.201960 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:44.201927 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" event={"ID":"5ba0da2c-9080-4a7e-b955-e1e2640a45b7","Type":"ContainerStarted","Data":"9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1"} Jul 08 20:35:44.789326 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:44.789279 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bccea67-8ce3-46db-a616-860433e01394" path="/var/lib/kubelet/pods/9bccea67-8ce3-46db-a616-860433e01394/volumes" Jul 08 20:35:44.805909 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:44.805886 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:44.806118 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:44.806100 2573 patch_prober.go:28] interesting pod/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 08 20:35:44.806206 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:44.806145 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 08 20:35:45.805887 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:45.805851 2573 patch_prober.go:28] interesting pod/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 08 20:35:45.806342 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:45.805898 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 08 20:35:46.806004 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:46.805971 2573 patch_prober.go:28] interesting pod/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 08 20:35:46.806503 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:46.806021 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 08 20:35:47.805898 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:47.805867 2573 patch_prober.go:28] interesting pod/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 08 20:35:47.806084 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:47.805920 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 08 20:35:48.809967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:48.809941 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:48.810403 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:48.810173 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:48.810805 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:48.810788 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:35:48.899107 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:48.899082 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:48.999034 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:48.999008 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:49.203221 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:49.203193 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf"] Jul 08 20:35:49.203430 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:49.203406 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" containerID="cri-o://5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8" gracePeriod=30 Jul 08 20:35:49.215282 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:49.215261 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:49.667650 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:49.667626 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:50.095531 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.095443 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:50.204907 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.204876 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:50.306952 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.306926 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:50.434173 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.434144 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq"] Jul 08 20:35:50.434460 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.434435 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" podUID="0a538285-e988-4d96-937c-541741bd3c65" containerName="maas-api" containerID="cri-o://d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59" gracePeriod=30 Jul 08 20:35:50.468416 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.468374 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:50.672135 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.672115 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:50.735295 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.735244 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a538285-e988-4d96-937c-541741bd3c65-maas-api-tls\") pod \"0a538285-e988-4d96-937c-541741bd3c65\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " Jul 08 20:35:50.735295 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.735286 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a538285-e988-4d96-937c-541741bd3c65-openshift-service-ca\") pod \"0a538285-e988-4d96-937c-541741bd3c65\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " Jul 08 20:35:50.735452 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.735327 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhmd8\" (UniqueName: \"kubernetes.io/projected/0a538285-e988-4d96-937c-541741bd3c65-kube-api-access-dhmd8\") pod \"0a538285-e988-4d96-937c-541741bd3c65\" (UID: \"0a538285-e988-4d96-937c-541741bd3c65\") " Jul 08 20:35:50.735588 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.735571 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a538285-e988-4d96-937c-541741bd3c65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a538285-e988-4d96-937c-541741bd3c65" (UID: "0a538285-e988-4d96-937c-541741bd3c65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:35:50.737216 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.737193 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a538285-e988-4d96-937c-541741bd3c65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a538285-e988-4d96-937c-541741bd3c65" (UID: "0a538285-e988-4d96-937c-541741bd3c65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:35:50.737344 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.737280 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a538285-e988-4d96-937c-541741bd3c65-kube-api-access-dhmd8" (OuterVolumeSpecName: "kube-api-access-dhmd8") pod "0a538285-e988-4d96-937c-541741bd3c65" (UID: "0a538285-e988-4d96-937c-541741bd3c65"). InnerVolumeSpecName "kube-api-access-dhmd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:35:50.836564 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.836541 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a538285-e988-4d96-937c-541741bd3c65-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:50.836564 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.836563 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a538285-e988-4d96-937c-541741bd3c65-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:50.836687 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:50.836571 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhmd8\" (UniqueName: \"kubernetes.io/projected/0a538285-e988-4d96-937c-541741bd3c65-kube-api-access-dhmd8\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:51.238501 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:51.238467 2573 generic.go:356] "Generic (PLEG): container finished" podID="0a538285-e988-4d96-937c-541741bd3c65" containerID="d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59" exitCode=0 Jul 08 20:35:51.238912 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:51.238510 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" Jul 08 20:35:51.238912 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:51.238548 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" event={"ID":"0a538285-e988-4d96-937c-541741bd3c65","Type":"ContainerDied","Data":"d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59"} Jul 08 20:35:51.238912 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:51.238596 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq" event={"ID":"0a538285-e988-4d96-937c-541741bd3c65","Type":"ContainerDied","Data":"efc5a6fcfa8f84116e2b2a9eae8e314b27fff3149ba96e1c4b2927460a43123c"} Jul 08 20:35:51.238912 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:51.238665 2573 scope.go:117] "RemoveContainer" containerID="d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59" Jul 08 20:35:51.247258 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:51.247245 2573 scope.go:117] "RemoveContainer" containerID="d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59" Jul 08 20:35:51.247525 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:51.247509 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59\": container with ID starting with d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59 not found: ID does not exist" containerID="d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59" Jul 08 20:35:51.247594 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:51.247533 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59"} err="failed to get container status \"d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59\": rpc error: code = NotFound desc = could not find container \"d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59\": container with ID starting with d8fbdad829a6ea60c22e53314e03d10c6c3b28e2a7c9b7ce10be2a530fb98d59 not found: ID does not exist" Jul 08 20:35:51.257171 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:51.257149 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq"] Jul 08 20:35:51.259257 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:51.259238 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2f232c-f47c96779-sb8pq"] Jul 08 20:35:52.786892 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:52.786861 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a538285-e988-4d96-937c-541741bd3c65" path="/var/lib/kubelet/pods/0a538285-e988-4d96-937c-541741bd3c65/volumes" Jul 08 20:35:54.450934 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.450913 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:54.470795 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.470744 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-data\") pod \"8e7128e6-0c54-4727-87cc-39be90e600e4\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " Jul 08 20:35:54.470795 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.470787 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-token\") pod \"8e7128e6-0c54-4727-87cc-39be90e600e4\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " Jul 08 20:35:54.470955 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.470819 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-certs\") pod \"8e7128e6-0c54-4727-87cc-39be90e600e4\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " Jul 08 20:35:54.470955 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.470840 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fzf8\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-kube-api-access-5fzf8\") pod \"8e7128e6-0c54-4727-87cc-39be90e600e4\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " Jul 08 20:35:54.470955 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.470855 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-credential-socket\") pod \"8e7128e6-0c54-4727-87cc-39be90e600e4\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " Jul 08 20:35:54.470955 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.470885 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7128e6-0c54-4727-87cc-39be90e600e4-istiod-ca-cert\") pod \"8e7128e6-0c54-4727-87cc-39be90e600e4\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " Jul 08 20:35:54.470955 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.470916 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-podinfo\") pod \"8e7128e6-0c54-4727-87cc-39be90e600e4\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " Jul 08 20:35:54.471214 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471010 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-socket\") pod \"8e7128e6-0c54-4727-87cc-39be90e600e4\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " Jul 08 20:35:54.471214 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471039 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-envoy\") pod \"8e7128e6-0c54-4727-87cc-39be90e600e4\" (UID: \"8e7128e6-0c54-4727-87cc-39be90e600e4\") " Jul 08 20:35:54.471214 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471087 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e7128e6-0c54-4727-87cc-39be90e600e4" (UID: "8e7128e6-0c54-4727-87cc-39be90e600e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:54.471405 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471284 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e7128e6-0c54-4727-87cc-39be90e600e4" (UID: "8e7128e6-0c54-4727-87cc-39be90e600e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:54.471405 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471338 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7128e6-0c54-4727-87cc-39be90e600e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e7128e6-0c54-4727-87cc-39be90e600e4" (UID: "8e7128e6-0c54-4727-87cc-39be90e600e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:35:54.471516 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471410 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:54.471516 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471428 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7128e6-0c54-4727-87cc-39be90e600e4-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:54.471516 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471440 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:54.471675 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471535 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e7128e6-0c54-4727-87cc-39be90e600e4" (UID: "8e7128e6-0c54-4727-87cc-39be90e600e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:54.471675 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.471651 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e7128e6-0c54-4727-87cc-39be90e600e4" (UID: "8e7128e6-0c54-4727-87cc-39be90e600e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:54.473931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.473907 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e7128e6-0c54-4727-87cc-39be90e600e4" (UID: "8e7128e6-0c54-4727-87cc-39be90e600e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:35:54.473931 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.473908 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e7128e6-0c54-4727-87cc-39be90e600e4" (UID: "8e7128e6-0c54-4727-87cc-39be90e600e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:35:54.474085 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.473945 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e7128e6-0c54-4727-87cc-39be90e600e4" (UID: "8e7128e6-0c54-4727-87cc-39be90e600e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:35:54.474174 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.474157 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-kube-api-access-5fzf8" (OuterVolumeSpecName: "kube-api-access-5fzf8") pod "8e7128e6-0c54-4727-87cc-39be90e600e4" (UID: "8e7128e6-0c54-4727-87cc-39be90e600e4"). InnerVolumeSpecName "kube-api-access-5fzf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:35:54.571889 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.571862 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:54.571889 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.571886 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:54.572043 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.571896 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fzf8\" (UniqueName: \"kubernetes.io/projected/8e7128e6-0c54-4727-87cc-39be90e600e4-kube-api-access-5fzf8\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:54.572043 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.571905 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:54.572043 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.571914 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:54.572043 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:54.571924 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7128e6-0c54-4727-87cc-39be90e600e4-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:35:55.260580 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.260545 2573 generic.go:356] "Generic (PLEG): container finished" podID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerID="5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8" exitCode=0 Jul 08 20:35:55.260739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.260595 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" Jul 08 20:35:55.260739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.260593 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" event={"ID":"8e7128e6-0c54-4727-87cc-39be90e600e4","Type":"ContainerDied","Data":"5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8"} Jul 08 20:35:55.260739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.260712 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf" event={"ID":"8e7128e6-0c54-4727-87cc-39be90e600e4","Type":"ContainerDied","Data":"4729013454cf63546fce999fdea546067c4f68427ebe6bb8038a3963c772d534"} Jul 08 20:35:55.260862 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.260741 2573 scope.go:117] "RemoveContainer" containerID="5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8" Jul 08 20:35:55.269410 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.269394 2573 scope.go:117] "RemoveContainer" containerID="97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b" Jul 08 20:35:55.277874 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.277853 2573 scope.go:117] "RemoveContainer" containerID="5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8" Jul 08 20:35:55.278197 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:55.278176 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8\": container with ID starting with 5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8 not found: ID does not exist" containerID="5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8" Jul 08 20:35:55.278278 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.278206 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8"} err="failed to get container status \"5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8\": rpc error: code = NotFound desc = could not find container \"5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8\": container with ID starting with 5f1cd43da6390be0e6bd59ed70c45df0c7254046865524df2bcef9b94e2816f8 not found: ID does not exist" Jul 08 20:35:55.278278 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.278222 2573 scope.go:117] "RemoveContainer" containerID="97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b" Jul 08 20:35:55.278578 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:35:55.278554 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b\": container with ID starting with 97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b not found: ID does not exist" containerID="97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b" Jul 08 20:35:55.278629 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.278584 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b"} err="failed to get container status \"97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b\": rpc error: code = NotFound desc = could not find container \"97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b\": container with ID starting with 97bac0062e241201ac305144da0451ade7f1294fa91f2fdf3e043364674e7e8b not found: ID does not exist" Jul 08 20:35:55.278960 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.278944 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf"] Jul 08 20:35:55.286118 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:55.286096 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-575204-openshift-default-5579cd8f85-9xmzf"] Jul 08 20:35:56.252993 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:56.252944 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:56.369116 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:56.369086 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:56.526898 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:56.526828 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws"] Jul 08 20:35:56.527039 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:56.527022 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" containerID="cri-o://9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1" gracePeriod=30 Jul 08 20:35:56.549598 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:56.549569 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:56.787547 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:56.787491 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" path="/var/lib/kubelet/pods/8e7128e6-0c54-4727-87cc-39be90e600e4/volumes" Jul 08 20:35:57.160125 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.160048 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:57.485847 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.485819 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6"] Jul 08 20:35:57.486209 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486196 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" Jul 08 20:35:57.486251 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486210 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" Jul 08 20:35:57.486251 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486222 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a538285-e988-4d96-937c-541741bd3c65" containerName="maas-api" Jul 08 20:35:57.486251 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486228 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a538285-e988-4d96-937c-541741bd3c65" containerName="maas-api" Jul 08 20:35:57.486251 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486234 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" Jul 08 20:35:57.486251 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486240 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" Jul 08 20:35:57.486417 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486261 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bccea67-8ce3-46db-a616-860433e01394" containerName="maas-api" Jul 08 20:35:57.486417 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486266 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bccea67-8ce3-46db-a616-860433e01394" containerName="maas-api" Jul 08 20:35:57.486417 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486339 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a538285-e988-4d96-937c-541741bd3c65" containerName="maas-api" Jul 08 20:35:57.486417 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486351 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" Jul 08 20:35:57.486417 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486356 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bccea67-8ce3-46db-a616-860433e01394" containerName="maas-api" Jul 08 20:35:57.486417 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.486364 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7128e6-0c54-4727-87cc-39be90e600e4" containerName="istio-proxy" Jul 08 20:35:57.489224 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.489208 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.492446 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.492427 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ec6309-openshift-default-dockercfg-rqt2n\"" Jul 08 20:35:57.506956 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.506933 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6"] Jul 08 20:35:57.515893 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.515869 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:57.595504 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.595473 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq9br\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-kube-api-access-gq9br\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.595657 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.595509 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7e81f8-2934-4a71-a12c-2ad946dcf626-istiod-ca-cert\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.595657 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.595552 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-credential-socket\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.595657 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.595629 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-socket\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.595802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.595669 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-token\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.595802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.595696 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-envoy\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.595802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.595750 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-certs\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.595802 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.595789 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-podinfo\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.595944 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.595816 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-data\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.696695 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.696671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-socket\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.696796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.696706 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-token\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.696796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.696724 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-envoy\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.696796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.696738 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-certs\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.696796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.696764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-podinfo\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.696796 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.696793 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-data\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.697053 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.696824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq9br\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-kube-api-access-gq9br\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.697053 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.696848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7e81f8-2934-4a71-a12c-2ad946dcf626-istiod-ca-cert\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.697053 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.697034 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-credential-socket\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.697206 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.697086 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-socket\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.697206 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.697150 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-certs\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.697333 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.697230 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-data\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.697445 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.697426 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-credential-socket\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.697820 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.697797 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7e81f8-2934-4a71-a12c-2ad946dcf626-istiod-ca-cert\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.698836 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.698817 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-envoy\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.699229 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.699209 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-podinfo\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.705458 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.705440 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-token\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.705593 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.705578 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq9br\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-kube-api-access-gq9br\") pod \"e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.798594 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.798524 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:57.935088 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.935063 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6"] Jul 08 20:35:57.936232 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:35:57.936212 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7e81f8_2934_4a71_a12c_2ad946dcf626.slice/crio-341774b02f20f2ae048011395ef557dc2daae39952286368fe1785efb5d25157 WatchSource:0}: Error finding container 341774b02f20f2ae048011395ef557dc2daae39952286368fe1785efb5d25157: Status 404 returned error can't find the container with id 341774b02f20f2ae048011395ef557dc2daae39952286368fe1785efb5d25157 Jul 08 20:35:57.938203 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.938170 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:57.938264 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.938250 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:57.938323 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:57.938294 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:35:58.000670 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:58.000645 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:35:58.278819 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:58.278788 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" event={"ID":"de7e81f8-2934-4a71-a12c-2ad946dcf626","Type":"ContainerStarted","Data":"b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d"} Jul 08 20:35:58.279006 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:58.278836 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" event={"ID":"de7e81f8-2934-4a71-a12c-2ad946dcf626","Type":"ContainerStarted","Data":"341774b02f20f2ae048011395ef557dc2daae39952286368fe1785efb5d25157"} Jul 08 20:35:58.305813 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:58.305771 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" podStartSLOduration=1.305756869 podStartE2EDuration="1.305756869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:35:58.303699266 +0000 UTC m=+2126.300716205" watchObservedRunningTime="2026-07-08 20:35:58.305756869 +0000 UTC m=+2126.302773804" Jul 08 20:35:58.798995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:58.798968 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:58.803673 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:58.803653 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:59.285113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:59.285081 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:35:59.286058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:35:59.286041 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:36:00.278668 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.278635 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:00.366651 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.366624 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m"] Jul 08 20:36:00.368920 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.368900 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:00.369273 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.369260 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:00.372030 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.372009 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 20:36:00.372125 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.372077 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 20:36:00.373208 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.373194 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ec6309-serving-cert\"" Jul 08 20:36:00.373305 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.373197 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x22jf\"" Jul 08 20:36:00.384341 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.384299 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m"] Jul 08 20:36:00.420962 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.420940 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwj99\" (UniqueName: \"kubernetes.io/projected/536fc28c-1090-42b2-bf25-9970898be417-kube-api-access-nwj99\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:00.421051 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.420974 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536fc28c-1090-42b2-bf25-9970898be417-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:00.421105 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.421051 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536fc28c-1090-42b2-bf25-9970898be417-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:00.450483 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.450458 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:00.522204 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.522175 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536fc28c-1090-42b2-bf25-9970898be417-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:00.522357 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.522239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwj99\" (UniqueName: \"kubernetes.io/projected/536fc28c-1090-42b2-bf25-9970898be417-kube-api-access-nwj99\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:00.522357 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.522266 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536fc28c-1090-42b2-bf25-9970898be417-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:00.522473 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:36:00.522457 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-serving-cert: secret "maas-api-e2e-shared-a-ec6309-serving-cert" not found Jul 08 20:36:00.522547 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:36:00.522518 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/536fc28c-1090-42b2-bf25-9970898be417-maas-api-tls podName:536fc28c-1090-42b2-bf25-9970898be417 nodeName:}" failed. No retries permitted until 2026-07-08 20:36:01.022501743 +0000 UTC m=+2129.019518657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/536fc28c-1090-42b2-bf25-9970898be417-maas-api-tls") pod "maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" (UID: "536fc28c-1090-42b2-bf25-9970898be417") : secret "maas-api-e2e-shared-a-ec6309-serving-cert" not found Jul 08 20:36:00.522874 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.522854 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536fc28c-1090-42b2-bf25-9970898be417-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:00.532079 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.532026 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwj99\" (UniqueName: \"kubernetes.io/projected/536fc28c-1090-42b2-bf25-9970898be417-kube-api-access-nwj99\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:00.614136 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.614117 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:00.975724 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:00.975701 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:01.027820 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.027799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536fc28c-1090-42b2-bf25-9970898be417-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:01.030169 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.030150 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536fc28c-1090-42b2-bf25-9970898be417-maas-api-tls\") pod \"maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:01.075733 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.075709 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:01.281008 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.280933 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:01.411180 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.411155 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m"] Jul 08 20:36:01.412581 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:36:01.412553 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536fc28c_1090_42b2_bf25_9970898be417.slice/crio-9340af5e432c9a9f7dfa3c2f302ec7af8fa0882c05ee77b89ddbe22b9e0452c2 WatchSource:0}: Error finding container 9340af5e432c9a9f7dfa3c2f302ec7af8fa0882c05ee77b89ddbe22b9e0452c2: Status 404 returned error can't find the container with id 9340af5e432c9a9f7dfa3c2f302ec7af8fa0882c05ee77b89ddbe22b9e0452c2 Jul 08 20:36:01.847003 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.846978 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:36:01.935113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935093 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zsrh\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-kube-api-access-7zsrh\") pod \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " Jul 08 20:36:01.935245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935126 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istiod-ca-cert\") pod \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " Jul 08 20:36:01.935245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935145 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-certs\") pod \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " Jul 08 20:36:01.935245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935162 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-credential-socket\") pod \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " Jul 08 20:36:01.935245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935185 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-token\") pod \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " Jul 08 20:36:01.935489 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935258 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-socket\") pod \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " Jul 08 20:36:01.935489 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935332 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-podinfo\") pod \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " Jul 08 20:36:01.935489 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935398 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-data\") pod \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " Jul 08 20:36:01.935489 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935420 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-envoy\") pod \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\" (UID: \"5ba0da2c-9080-4a7e-b955-e1e2640a45b7\") " Jul 08 20:36:01.935489 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935464 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ba0da2c-9080-4a7e-b955-e1e2640a45b7" (UID: "5ba0da2c-9080-4a7e-b955-e1e2640a45b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:36:01.935489 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935481 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ba0da2c-9080-4a7e-b955-e1e2640a45b7" (UID: "5ba0da2c-9080-4a7e-b955-e1e2640a45b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:36:01.935772 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935545 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ba0da2c-9080-4a7e-b955-e1e2640a45b7" (UID: "5ba0da2c-9080-4a7e-b955-e1e2640a45b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:36:01.935772 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935688 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ba0da2c-9080-4a7e-b955-e1e2640a45b7" (UID: "5ba0da2c-9080-4a7e-b955-e1e2640a45b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:36:01.935875 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935839 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:36:01.935875 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935861 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:36:01.935956 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935873 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:36:01.935956 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.935887 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:36:01.936149 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.936132 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ba0da2c-9080-4a7e-b955-e1e2640a45b7" (UID: "5ba0da2c-9080-4a7e-b955-e1e2640a45b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:36:01.937680 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.937661 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-kube-api-access-7zsrh" (OuterVolumeSpecName: "kube-api-access-7zsrh") pod "5ba0da2c-9080-4a7e-b955-e1e2640a45b7" (UID: "5ba0da2c-9080-4a7e-b955-e1e2640a45b7"). InnerVolumeSpecName "kube-api-access-7zsrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:36:01.937746 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.937688 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ba0da2c-9080-4a7e-b955-e1e2640a45b7" (UID: "5ba0da2c-9080-4a7e-b955-e1e2640a45b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:36:01.937921 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.937904 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ba0da2c-9080-4a7e-b955-e1e2640a45b7" (UID: "5ba0da2c-9080-4a7e-b955-e1e2640a45b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:36:01.937979 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:01.937916 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ba0da2c-9080-4a7e-b955-e1e2640a45b7" (UID: "5ba0da2c-9080-4a7e-b955-e1e2640a45b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:36:02.036536 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.036513 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:36:02.036536 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.036531 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:36:02.036536 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.036539 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:36:02.036700 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.036547 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zsrh\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-kube-api-access-7zsrh\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:36:02.036700 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.036555 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ba0da2c-9080-4a7e-b955-e1e2640a45b7-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:36:02.302447 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.302420 2573 generic.go:356] "Generic (PLEG): container finished" podID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerID="9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1" exitCode=0 Jul 08 20:36:02.302838 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.302462 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" Jul 08 20:36:02.302838 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.302494 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" event={"ID":"5ba0da2c-9080-4a7e-b955-e1e2640a45b7","Type":"ContainerDied","Data":"9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1"} Jul 08 20:36:02.302838 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.302540 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws" event={"ID":"5ba0da2c-9080-4a7e-b955-e1e2640a45b7","Type":"ContainerDied","Data":"87cc575e599f7c36d0f90f4c62b47181c07441f6ff5b6e967b6cc1ae8bcc09de"} Jul 08 20:36:02.302838 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.302567 2573 scope.go:117] "RemoveContainer" containerID="9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1" Jul 08 20:36:02.304246 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.304226 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" event={"ID":"536fc28c-1090-42b2-bf25-9970898be417","Type":"ContainerStarted","Data":"7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c"} Jul 08 20:36:02.304353 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.304265 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" event={"ID":"536fc28c-1090-42b2-bf25-9970898be417","Type":"ContainerStarted","Data":"9340af5e432c9a9f7dfa3c2f302ec7af8fa0882c05ee77b89ddbe22b9e0452c2"} Jul 08 20:36:02.304353 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.304291 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:02.312064 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.312048 2573 scope.go:117] "RemoveContainer" containerID="2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0" Jul 08 20:36:02.319789 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.319771 2573 scope.go:117] "RemoveContainer" containerID="9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1" Jul 08 20:36:02.320035 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:36:02.320019 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1\": container with ID starting with 9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1 not found: ID does not exist" containerID="9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1" Jul 08 20:36:02.320078 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.320044 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1"} err="failed to get container status \"9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1\": rpc error: code = NotFound desc = could not find container \"9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1\": container with ID starting with 9a2c70eeacdc6288683af8c29db13361c023ecccfbbc77ebae426d0c562021b1 not found: ID does not exist" Jul 08 20:36:02.320078 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.320061 2573 scope.go:117] "RemoveContainer" containerID="2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0" Jul 08 20:36:02.320280 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:36:02.320267 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0\": container with ID starting with 2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0 not found: ID does not exist" containerID="2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0" Jul 08 20:36:02.320342 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.320282 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0"} err="failed to get container status \"2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0\": rpc error: code = NotFound desc = could not find container \"2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0\": container with ID starting with 2fa68008ab2b627c5552d3e3364bbcecb4d651c97fcf321cb3630de249b320e0 not found: ID does not exist" Jul 08 20:36:02.325116 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.325072 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" podStartSLOduration=1.981186334 podStartE2EDuration="2.325056809s" totalImagesPullingTime="343.870475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:36:02.324844595 +0000 UTC m=+2130.321861532" watchObservedRunningTime="2026-07-08 20:36:02.325056809 +0000 UTC m=+2130.322073743" Jul 08 20:36:02.340131 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.340113 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws"] Jul 08 20:36:02.344727 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.344706 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2f232c-openshift-default-65575f7c89-x4jws"] Jul 08 20:36:02.786556 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:02.786530 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" path="/var/lib/kubelet/pods/5ba0da2c-9080-4a7e-b955-e1e2640a45b7/volumes" Jul 08 20:36:04.823356 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:04.823323 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:05.193462 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.193427 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:05.501419 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.501339 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66"] Jul 08 20:36:05.501959 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.501940 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" Jul 08 20:36:05.501959 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.501958 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" Jul 08 20:36:05.502106 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.501977 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" Jul 08 20:36:05.502106 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.501985 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" Jul 08 20:36:05.502106 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.502086 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" Jul 08 20:36:05.505503 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.505484 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.518862 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.518843 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-ca53e0-openshift-default-dockercfg-2qdbj\"" Jul 08 20:36:05.520069 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.520051 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66"] Jul 08 20:36:05.529847 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.529827 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:05.570739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.570713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mjj\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-kube-api-access-f9mjj\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.570885 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.570753 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-podinfo\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.570885 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.570770 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-token\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.570885 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.570793 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-certs\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.570885 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.570850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-credential-socket\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.571014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.570895 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-data\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.571014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.570927 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-socket\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.571014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.570971 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-envoy\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.571014 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.570989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istiod-ca-cert\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.630257 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.630230 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:05.672117 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672089 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mjj\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-kube-api-access-f9mjj\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672297 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672131 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-podinfo\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672297 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672147 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-token\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672297 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-certs\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672297 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672188 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-credential-socket\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672297 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-data\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672589 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-socket\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672589 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672487 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-envoy\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672589 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istiod-ca-cert\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672741 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672634 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-credential-socket\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672741 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672657 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-data\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672842 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672805 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-socket\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.672995 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.672974 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-certs\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.673154 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.673138 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istiod-ca-cert\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.674956 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.674924 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-envoy\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.675071 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.675008 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-podinfo\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.680689 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.680670 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mjj\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-kube-api-access-f9mjj\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.680891 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.680871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-token\") pod \"e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.815178 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.815082 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:05.951458 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.951429 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66"] Jul 08 20:36:05.954502 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:36:05.954474 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda73a27f9_33a1_493e_9ca0_0af5bb8d5cb1.slice/crio-73cd4778edc03729c30885ec6b786e62a40cc4474e533c6987cc19e76449e6f8 WatchSource:0}: Error finding container 73cd4778edc03729c30885ec6b786e62a40cc4474e533c6987cc19e76449e6f8: Status 404 returned error can't find the container with id 73cd4778edc03729c30885ec6b786e62a40cc4474e533c6987cc19e76449e6f8 Jul 08 20:36:05.956954 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.956924 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:36:05.957062 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.956989 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:36:05.957062 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:05.957018 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:36:06.326696 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:06.326661 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" event={"ID":"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1","Type":"ContainerStarted","Data":"10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa"} Jul 08 20:36:06.326908 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:06.326708 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" event={"ID":"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1","Type":"ContainerStarted","Data":"73cd4778edc03729c30885ec6b786e62a40cc4474e533c6987cc19e76449e6f8"} Jul 08 20:36:06.349679 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:06.349605 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" podStartSLOduration=1.349586491 podStartE2EDuration="1.349586491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:36:06.348998337 +0000 UTC m=+2134.346015274" watchObservedRunningTime="2026-07-08 20:36:06.349586491 +0000 UTC m=+2134.346603427" Jul 08 20:36:06.816136 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:06.816093 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:07.820397 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:07.820371 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:08.314538 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.314513 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:36:08.344630 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.344607 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:08.345755 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.345730 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:08.488528 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.488496 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:08.517289 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.517256 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz"] Jul 08 20:36:08.517816 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.517798 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ba0da2c-9080-4a7e-b955-e1e2640a45b7" containerName="istio-proxy" Jul 08 20:36:08.519894 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.519876 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:08.522827 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.522809 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ca53e0-serving-cert\"" Jul 08 20:36:08.531221 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.531200 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz"] Jul 08 20:36:08.590177 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.590106 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:08.600009 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.599986 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-maas-api-tls\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:08.600113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.600089 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-kube-api-access-m484k\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:08.600152 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.600129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:08.701201 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.701178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-kube-api-access-m484k\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:08.701383 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.701212 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:08.701383 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.701245 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-maas-api-tls\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:08.701512 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:36:08.701408 2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-serving-cert: secret "maas-api-e2e-shared-b-ca53e0-serving-cert" not found Jul 08 20:36:08.701512 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:36:08.701470 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-maas-api-tls podName:7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e nodeName:}" failed. No retries permitted until 2026-07-08 20:36:09.201452333 +0000 UTC m=+2137.198469254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-maas-api-tls") pod "maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" (UID: "7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e") : secret "maas-api-e2e-shared-b-ca53e0-serving-cert" not found Jul 08 20:36:08.701887 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.701867 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:08.711780 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.711758 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-kube-api-access-m484k\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:08.850482 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:08.850420 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:09.138709 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:09.138630 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:09.205285 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:09.205255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-maas-api-tls\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:09.207897 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:09.207870 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-maas-api-tls\") pod \"maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:09.237573 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:09.237548 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:09.431291 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:09.431252 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:09.605700 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:09.605676 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz"] Jul 08 20:36:09.607295 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:36:09.607274 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa7edf6_85fb_4fe8_bd34_9d0d41407d8e.slice/crio-b77a7c3417fa8e71255f0bc184007a32a820aecae107d3602e71a573adb1cc50 WatchSource:0}: Error finding container b77a7c3417fa8e71255f0bc184007a32a820aecae107d3602e71a573adb1cc50: Status 404 returned error can't find the container with id b77a7c3417fa8e71255f0bc184007a32a820aecae107d3602e71a573adb1cc50 Jul 08 20:36:10.369449 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:10.369407 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" event={"ID":"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e","Type":"ContainerStarted","Data":"e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d"} Jul 08 20:36:10.369836 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:10.369490 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" event={"ID":"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e","Type":"ContainerStarted","Data":"b77a7c3417fa8e71255f0bc184007a32a820aecae107d3602e71a573adb1cc50"} Jul 08 20:36:10.369836 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:10.369525 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:10.387771 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:10.387730 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" podStartSLOduration=1.845621001 podStartE2EDuration="2.387712589s" totalImagesPullingTime="542.091588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:36:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:36:10.386377689 +0000 UTC m=+2138.383394624" watchObservedRunningTime="2026-07-08 20:36:10.387712589 +0000 UTC m=+2138.384729526" Jul 08 20:36:10.507570 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:10.507534 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:12.299580 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:12.299543 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:13.029185 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:13.029149 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:15.328451 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:15.328420 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 08 20:36:15.328850 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:15.328475 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 08 20:36:16.379552 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:16.379489 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:36:16.404366 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:16.404335 2573 generic.go:356] "Generic (PLEG): container finished" podID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerID="b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d" exitCode=0 Jul 08 20:36:16.404497 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:16.404403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" event={"ID":"de7e81f8-2934-4a71-a12c-2ad946dcf626","Type":"ContainerDied","Data":"b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d"} Jul 08 20:36:16.404809 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:16.404793 2573 scope.go:117] "RemoveContainer" containerID="b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d" Jul 08 20:36:16.405763 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:16.405742 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:36:16.405834 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:16.405822 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:36:16.405872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:16.405861 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:36:17.413542 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:17.413507 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" event={"ID":"de7e81f8-2934-4a71-a12c-2ad946dcf626","Type":"ContainerStarted","Data":"bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9"} Jul 08 20:36:17.799002 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:17.798971 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:36:17.799184 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:17.799062 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 08 20:36:17.799184 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:17.799096 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 08 20:36:18.799935 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:18.799898 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 08 20:36:18.800364 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:18.799945 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 08 20:36:19.799700 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:19.799662 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 08 20:36:19.799879 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:19.799718 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 08 20:36:20.799286 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:20.799253 2573 patch_prober.go:28] interesting pod/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 08 20:36:20.799674 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:20.799320 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 08 20:36:21.748567 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:21.748526 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:21.803701 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:21.803678 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:36:21.804054 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:21.803902 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:36:21.804470 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:21.804453 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:36:21.831149 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:21.831124 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:23.345778 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:23.345749 2573 patch_prober.go:28] interesting pod/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 08 20:36:23.346174 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:23.345796 2573 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 08 20:36:23.445292 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:23.445268 2573 generic.go:356] "Generic (PLEG): container finished" podID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerID="10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa" exitCode=0 Jul 08 20:36:23.445432 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:23.445341 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" event={"ID":"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1","Type":"ContainerDied","Data":"10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa"} Jul 08 20:36:23.445872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:23.445857 2573 scope.go:117] "RemoveContainer" containerID="10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa" Jul 08 20:36:23.446732 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:23.446711 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:36:23.446786 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:23.446778 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:36:23.446821 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:23.446808 2573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 08 20:36:24.452580 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:24.452501 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" event={"ID":"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1","Type":"ContainerStarted","Data":"3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827"} Jul 08 20:36:24.816060 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:24.815969 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:24.816245 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:24.816223 2573 patch_prober.go:28] interesting pod/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 08 20:36:24.816353 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:24.816292 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 08 20:36:25.815692 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:25.815662 2573 patch_prober.go:28] interesting pod/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 08 20:36:25.816059 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:25.815712 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 08 20:36:26.815437 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:26.815406 2573 patch_prober.go:28] interesting pod/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 08 20:36:26.815622 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:26.815464 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 08 20:36:27.815960 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:27.815927 2573 patch_prober.go:28] interesting pod/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 08 20:36:27.816391 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:27.815980 2573 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 08 20:36:28.820002 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:28.819976 2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:28.820369 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:28.820339 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:28.821012 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:28.820991 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:36:30.013973 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:30.013941 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:36:30.180877 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:36:30.180841 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:37:27.045172 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:37:27.045138 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:38:13.899546 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:38:13.899512 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:18.419708 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:18.419669 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:19.104388 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.104356 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m"] Jul 08 20:39:19.104618 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.104594 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" podUID="536fc28c-1090-42b2-bf25-9970898be417" containerName="maas-api" containerID="cri-o://7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c" gracePeriod=30 Jul 08 20:39:19.352932 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.352909 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:39:19.362337 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.362257 2573 generic.go:356] "Generic (PLEG): container finished" podID="536fc28c-1090-42b2-bf25-9970898be417" containerID="7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c" exitCode=0 Jul 08 20:39:19.362337 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.362326 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" Jul 08 20:39:19.362485 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.362348 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" event={"ID":"536fc28c-1090-42b2-bf25-9970898be417","Type":"ContainerDied","Data":"7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c"} Jul 08 20:39:19.362485 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.362403 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m" event={"ID":"536fc28c-1090-42b2-bf25-9970898be417","Type":"ContainerDied","Data":"9340af5e432c9a9f7dfa3c2f302ec7af8fa0882c05ee77b89ddbe22b9e0452c2"} Jul 08 20:39:19.362485 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.362428 2573 scope.go:117] "RemoveContainer" containerID="7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c" Jul 08 20:39:19.372323 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.371781 2573 scope.go:117] "RemoveContainer" containerID="7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c" Jul 08 20:39:19.372323 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:39:19.372125 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c\": container with ID starting with 7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c not found: ID does not exist" containerID="7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c" Jul 08 20:39:19.372323 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.372157 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c"} err="failed to get container status \"7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c\": rpc error: code = NotFound desc = could not find container \"7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c\": container with ID starting with 7074f2914ba04f75f9b5b75f64f15555925509a2537714f3763c8f4e1360bc9c not found: ID does not exist" Jul 08 20:39:19.432360 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.432299 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536fc28c-1090-42b2-bf25-9970898be417-openshift-service-ca\") pod \"536fc28c-1090-42b2-bf25-9970898be417\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " Jul 08 20:39:19.432805 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.432458 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwj99\" (UniqueName: \"kubernetes.io/projected/536fc28c-1090-42b2-bf25-9970898be417-kube-api-access-nwj99\") pod \"536fc28c-1090-42b2-bf25-9970898be417\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " Jul 08 20:39:19.432805 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.432565 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536fc28c-1090-42b2-bf25-9970898be417-maas-api-tls\") pod \"536fc28c-1090-42b2-bf25-9970898be417\" (UID: \"536fc28c-1090-42b2-bf25-9970898be417\") " Jul 08 20:39:19.432805 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.432634 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536fc28c-1090-42b2-bf25-9970898be417-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "536fc28c-1090-42b2-bf25-9970898be417" (UID: "536fc28c-1090-42b2-bf25-9970898be417"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:39:19.432974 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.432893 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/536fc28c-1090-42b2-bf25-9970898be417-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:19.434619 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.434597 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536fc28c-1090-42b2-bf25-9970898be417-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "536fc28c-1090-42b2-bf25-9970898be417" (UID: "536fc28c-1090-42b2-bf25-9970898be417"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:39:19.434915 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.434899 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536fc28c-1090-42b2-bf25-9970898be417-kube-api-access-nwj99" (OuterVolumeSpecName: "kube-api-access-nwj99") pod "536fc28c-1090-42b2-bf25-9970898be417" (UID: "536fc28c-1090-42b2-bf25-9970898be417"). InnerVolumeSpecName "kube-api-access-nwj99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:39:19.534238 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.534202 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwj99\" (UniqueName: \"kubernetes.io/projected/536fc28c-1090-42b2-bf25-9970898be417-kube-api-access-nwj99\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:19.534238 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.534231 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/536fc28c-1090-42b2-bf25-9970898be417-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:19.648481 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.648403 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:19.684334 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.684293 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m"] Jul 08 20:39:19.688255 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:19.688231 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ec6309-689c8b4d85-46h2m"] Jul 08 20:39:20.787330 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:20.787282 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="536fc28c-1090-42b2-bf25-9970898be417" path="/var/lib/kubelet/pods/536fc28c-1090-42b2-bf25-9970898be417/volumes" Jul 08 20:39:24.586662 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:24.586580 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:24.641788 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:24.641759 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:24.841260 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:24.841164 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6"] Jul 08 20:39:24.841454 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:24.841420 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" containerID="cri-o://bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9" gracePeriod=30 Jul 08 20:39:24.859133 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:24.859100 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:25.283496 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:25.283465 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:25.821563 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:25.821530 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:25.911859 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:25.911831 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:26.038721 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.038687 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz"] Jul 08 20:39:26.038929 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.038897 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" podUID="7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e" containerName="maas-api" containerID="cri-o://e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d" gracePeriod=30 Jul 08 20:39:26.072575 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.072501 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:26.285575 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.285554 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:39:26.398707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.398618 2573 generic.go:356] "Generic (PLEG): container finished" podID="7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e" containerID="e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d" exitCode=0 Jul 08 20:39:26.398707 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.398664 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" Jul 08 20:39:26.398896 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.398709 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" event={"ID":"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e","Type":"ContainerDied","Data":"e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d"} Jul 08 20:39:26.398896 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.398760 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz" event={"ID":"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e","Type":"ContainerDied","Data":"b77a7c3417fa8e71255f0bc184007a32a820aecae107d3602e71a573adb1cc50"} Jul 08 20:39:26.398896 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.398781 2573 scope.go:117] "RemoveContainer" containerID="e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d" Jul 08 20:39:26.404994 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.404973 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-maas-api-tls\") pod \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " Jul 08 20:39:26.405109 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.405042 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-kube-api-access-m484k\") pod \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " Jul 08 20:39:26.405109 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.405073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-openshift-service-ca\") pod \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\" (UID: \"7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e\") " Jul 08 20:39:26.405473 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.405449 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e" (UID: "7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:39:26.406928 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.406899 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e" (UID: "7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 20:39:26.407034 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.407001 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-kube-api-access-m484k" (OuterVolumeSpecName: "kube-api-access-m484k") pod "7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e" (UID: "7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e"). InnerVolumeSpecName "kube-api-access-m484k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:39:26.407543 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.407528 2573 scope.go:117] "RemoveContainer" containerID="e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d" Jul 08 20:39:26.407804 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:39:26.407789 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d\": container with ID starting with e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d not found: ID does not exist" containerID="e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d" Jul 08 20:39:26.407848 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.407813 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d"} err="failed to get container status \"e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d\": rpc error: code = NotFound desc = could not find container \"e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d\": container with ID starting with e24f9677500555a05b56d3b17f22366e1079bc2c2f3bb54fa8486fa52b19a17d not found: ID does not exist" Jul 08 20:39:26.506449 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.506420 2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-maas-api-tls\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:26.506449 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.506446 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m484k\" (UniqueName: \"kubernetes.io/projected/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-kube-api-access-m484k\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:26.506449 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.506456 2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e-openshift-service-ca\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:26.720351 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.720322 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz"] Jul 08 20:39:26.724270 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.724244 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca53e0-59f7995f9-fzdnz"] Jul 08 20:39:26.787440 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:26.787407 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e" path="/var/lib/kubelet/pods/7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e/volumes" Jul 08 20:39:30.094917 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.094894 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:39:30.242668 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242643 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-socket\") pod \"de7e81f8-2934-4a71-a12c-2ad946dcf626\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " Jul 08 20:39:30.242872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242684 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-token\") pod \"de7e81f8-2934-4a71-a12c-2ad946dcf626\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " Jul 08 20:39:30.242872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242700 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-certs\") pod \"de7e81f8-2934-4a71-a12c-2ad946dcf626\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " Jul 08 20:39:30.242872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242741 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-envoy\") pod \"de7e81f8-2934-4a71-a12c-2ad946dcf626\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " Jul 08 20:39:30.242872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242760 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7e81f8-2934-4a71-a12c-2ad946dcf626-istiod-ca-cert\") pod \"de7e81f8-2934-4a71-a12c-2ad946dcf626\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " Jul 08 20:39:30.242872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242796 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-podinfo\") pod \"de7e81f8-2934-4a71-a12c-2ad946dcf626\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " Jul 08 20:39:30.242872 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242837 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-data\") pod \"de7e81f8-2934-4a71-a12c-2ad946dcf626\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " Jul 08 20:39:30.243278 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242872 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-credential-socket\") pod \"de7e81f8-2934-4a71-a12c-2ad946dcf626\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " Jul 08 20:39:30.243278 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242944 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "de7e81f8-2934-4a71-a12c-2ad946dcf626" (UID: "de7e81f8-2934-4a71-a12c-2ad946dcf626"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:30.243278 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.242955 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq9br\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-kube-api-access-gq9br\") pod \"de7e81f8-2934-4a71-a12c-2ad946dcf626\" (UID: \"de7e81f8-2934-4a71-a12c-2ad946dcf626\") " Jul 08 20:39:30.243278 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.243240 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7e81f8-2934-4a71-a12c-2ad946dcf626-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "de7e81f8-2934-4a71-a12c-2ad946dcf626" (UID: "de7e81f8-2934-4a71-a12c-2ad946dcf626"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:39:30.243561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.243349 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "de7e81f8-2934-4a71-a12c-2ad946dcf626" (UID: "de7e81f8-2934-4a71-a12c-2ad946dcf626"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:30.243561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.243443 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7e81f8-2934-4a71-a12c-2ad946dcf626-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:30.243561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.243465 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:30.243561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.243477 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:30.243561 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.243513 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "de7e81f8-2934-4a71-a12c-2ad946dcf626" (UID: "de7e81f8-2934-4a71-a12c-2ad946dcf626"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:30.243888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.243863 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-data" (OuterVolumeSpecName: "istio-data") pod "de7e81f8-2934-4a71-a12c-2ad946dcf626" (UID: "de7e81f8-2934-4a71-a12c-2ad946dcf626"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:30.245053 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.245031 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-token" (OuterVolumeSpecName: "istio-token") pod "de7e81f8-2934-4a71-a12c-2ad946dcf626" (UID: "de7e81f8-2934-4a71-a12c-2ad946dcf626"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:39:30.245165 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.245094 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-kube-api-access-gq9br" (OuterVolumeSpecName: "kube-api-access-gq9br") pod "de7e81f8-2934-4a71-a12c-2ad946dcf626" (UID: "de7e81f8-2934-4a71-a12c-2ad946dcf626"). InnerVolumeSpecName "kube-api-access-gq9br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:39:30.245390 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.245371 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "de7e81f8-2934-4a71-a12c-2ad946dcf626" (UID: "de7e81f8-2934-4a71-a12c-2ad946dcf626"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:30.245447 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.245423 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "de7e81f8-2934-4a71-a12c-2ad946dcf626" (UID: "de7e81f8-2934-4a71-a12c-2ad946dcf626"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:39:30.344139 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.344089 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:30.344139 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.344133 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:30.344139 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.344145 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:30.344139 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.344153 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7e81f8-2934-4a71-a12c-2ad946dcf626-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:30.344139 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.344164 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq9br\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-kube-api-access-gq9br\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:30.344498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.344172 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7e81f8-2934-4a71-a12c-2ad946dcf626-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:30.427989 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.427955 2573 generic.go:356] "Generic (PLEG): container finished" podID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerID="bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9" exitCode=0 Jul 08 20:39:30.428187 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.428009 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" Jul 08 20:39:30.428187 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.428031 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" event={"ID":"de7e81f8-2934-4a71-a12c-2ad946dcf626","Type":"ContainerDied","Data":"bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9"} Jul 08 20:39:30.428187 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.428078 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6" event={"ID":"de7e81f8-2934-4a71-a12c-2ad946dcf626","Type":"ContainerDied","Data":"341774b02f20f2ae048011395ef557dc2daae39952286368fe1785efb5d25157"} Jul 08 20:39:30.428187 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.428100 2573 scope.go:117] "RemoveContainer" containerID="bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9" Jul 08 20:39:30.437446 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.437428 2573 scope.go:117] "RemoveContainer" containerID="b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d" Jul 08 20:39:30.445210 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.445195 2573 scope.go:117] "RemoveContainer" containerID="bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9" Jul 08 20:39:30.445473 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:39:30.445453 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9\": container with ID starting with bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9 not found: ID does not exist" containerID="bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9" Jul 08 20:39:30.445562 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.445484 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9"} err="failed to get container status \"bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9\": rpc error: code = NotFound desc = could not find container \"bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9\": container with ID starting with bd643f3ac4bdcdd8364590a15e0fe3ffe8036064f02885b358abb133099879a9 not found: ID does not exist" Jul 08 20:39:30.445562 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.445506 2573 scope.go:117] "RemoveContainer" containerID="b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d" Jul 08 20:39:30.445743 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:39:30.445726 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d\": container with ID starting with b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d not found: ID does not exist" containerID="b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d" Jul 08 20:39:30.445788 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.445753 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d"} err="failed to get container status \"b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d\": rpc error: code = NotFound desc = could not find container \"b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d\": container with ID starting with b7a52ec4a9522bc4afa65023c0b198f1f8462b4a5d54fbbf205a9e9da289563d not found: ID does not exist" Jul 08 20:39:30.449765 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.449745 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6"] Jul 08 20:39:30.454858 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.454837 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ec6309-openshift-default-6656bf6dc9-9frn6"] Jul 08 20:39:30.787684 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:30.787656 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" path="/var/lib/kubelet/pods/de7e81f8-2934-4a71-a12c-2ad946dcf626/volumes" Jul 08 20:39:31.661058 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:31.661030 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:31.727069 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:31.727039 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:31.932921 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:31.932887 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66"] Jul 08 20:39:31.933144 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:31.933122 2573 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" containerID="cri-o://3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827" gracePeriod=30 Jul 08 20:39:31.958498 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:31.958472 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:37.178046 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.178022 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:39:37.212980 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.212878 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9mjj\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-kube-api-access-f9mjj\") pod \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " Jul 08 20:39:37.212980 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.212933 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-podinfo\") pod \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " Jul 08 20:39:37.212980 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.212971 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-credential-socket\") pod \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " Jul 08 20:39:37.213267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213036 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istiod-ca-cert\") pod \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " Jul 08 20:39:37.213267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213073 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-certs\") pod \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " Jul 08 20:39:37.213267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213123 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-socket\") pod \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " Jul 08 20:39:37.213267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213180 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-data\") pod \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " Jul 08 20:39:37.213267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213210 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-token\") pod \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " Jul 08 20:39:37.213267 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213236 2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-envoy\") pod \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\" (UID: \"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1\") " Jul 08 20:39:37.213611 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213361 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" (UID: "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:37.213720 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213601 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" (UID: "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 20:39:37.213720 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213636 2573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-credential-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:37.213993 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.213932 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" (UID: "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:37.214174 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.214142 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-data" (OuterVolumeSpecName: "istio-data") pod "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" (UID: "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:37.214352 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.214205 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" (UID: "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:37.215602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.215560 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-kube-api-access-f9mjj" (OuterVolumeSpecName: "kube-api-access-f9mjj") pod "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" (UID: "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1"). InnerVolumeSpecName "kube-api-access-f9mjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:39:37.215602 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.215573 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-token" (OuterVolumeSpecName: "istio-token") pod "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" (UID: "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 20:39:37.215780 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.215645 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" (UID: "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 20:39:37.215983 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.215964 2573 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" (UID: "a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 20:39:37.314871 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.314837 2573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-data\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:37.314871 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.314865 2573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-token\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:37.314871 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.314875 2573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-envoy\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:37.315112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.314889 2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9mjj\" (UniqueName: \"kubernetes.io/projected/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-kube-api-access-f9mjj\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:37.315112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.314903 2573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istio-podinfo\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:37.315112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.314911 2573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-istiod-ca-cert\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:37.315112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.314918 2573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-certs\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:37.315112 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.314927 2573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1-workload-socket\") on node \"ip-10-0-132-18.ec2.internal\" DevicePath \"\"" Jul 08 20:39:37.463297 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.463209 2573 generic.go:356] "Generic (PLEG): container finished" podID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerID="3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827" exitCode=0 Jul 08 20:39:37.463297 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.463259 2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" Jul 08 20:39:37.463486 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.463297 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" event={"ID":"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1","Type":"ContainerDied","Data":"3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827"} Jul 08 20:39:37.463486 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.463370 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66" event={"ID":"a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1","Type":"ContainerDied","Data":"73cd4778edc03729c30885ec6b786e62a40cc4474e533c6987cc19e76449e6f8"} Jul 08 20:39:37.463486 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.463405 2573 scope.go:117] "RemoveContainer" containerID="3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827" Jul 08 20:39:37.476873 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.476843 2573 scope.go:117] "RemoveContainer" containerID="10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa" Jul 08 20:39:37.484959 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.484832 2573 scope.go:117] "RemoveContainer" containerID="3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827" Jul 08 20:39:37.485785 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:39:37.485758 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827\": container with ID starting with 3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827 not found: ID does not exist" containerID="3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827" Jul 08 20:39:37.485905 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.485795 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827"} err="failed to get container status \"3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827\": rpc error: code = NotFound desc = could not find container \"3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827\": container with ID starting with 3e329332af5f0fa78ed5fc9facc819fd6dc3f1d03865b9cd5cb10beb0cb9b827 not found: ID does not exist" Jul 08 20:39:37.485905 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.485818 2573 scope.go:117] "RemoveContainer" containerID="10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa" Jul 08 20:39:37.486124 ip-10-0-132-18 kubenswrapper[2573]: E0708 20:39:37.486102 2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa\": container with ID starting with 10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa not found: ID does not exist" containerID="10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa" Jul 08 20:39:37.486167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.486134 2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa"} err="failed to get container status \"10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa\": rpc error: code = NotFound desc = could not find container \"10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa\": container with ID starting with 10dfc8ad977e4833639b4442cc4893293c54074ee3a7fece601658b8283286fa not found: ID does not exist" Jul 08 20:39:37.487383 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.487362 2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66"] Jul 08 20:39:37.493975 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:37.493952 2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-ca53e0-openshift-default-57b976fb99-l5z66"] Jul 08 20:39:38.787228 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:38.787201 2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" path="/var/lib/kubelet/pods/a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1/volumes" Jul 08 20:39:39.273520 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:39.273490 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67959458dc-fdjcc_d9ecda20-8582-4251-83cb-706e1b13f347/copy-manifests/0.log" Jul 08 20:39:39.280719 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:39.280701 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67959458dc-fdjcc_d9ecda20-8582-4251-83cb-706e1b13f347/manager/0.log" Jul 08 20:39:41.238890 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:41.238865 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-p4x4k_fd398bc2-4cc4-4d1f-801f-08a7aa0322ca/manager/0.log" Jul 08 20:39:41.362167 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:41.362142 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pvzpn_7a4b6b9a-d0e7-4f22-9ded-94e70ed19098/kuadrant-console-plugin/0.log" Jul 08 20:39:42.774282 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:42.774251 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789db8bf69-ftszt_df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597/kube-auth-proxy/0.log" Jul 08 20:39:43.055956 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:43.055882 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-d9fpz_fa609f0b-a36a-45ea-a12c-4ad343179ff7/payload-pre-processing/0.log" Jul 08 20:39:43.756509 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:43.756482 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-5pl9n_abbf65ad-09da-489a-91b3-b8d241fdcc7b/main/0.log" Jul 08 20:39:44.275893 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:44.275869 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-brpzt_d69e6794-faa6-4a77-a58a-903281206b6a/main/0.log" Jul 08 20:39:44.419800 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:44.419777 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-rl7tj_8b852c64-d879-4a4a-9eb7-7535b98ab5bf/main/0.log" Jul 08 20:39:49.419343 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419296 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9gq7f/must-gather-nrdlq"] Jul 08 20:39:49.419888 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419867 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="536fc28c-1090-42b2-bf25-9970898be417" containerName="maas-api" Jul 08 20:39:49.419967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419891 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="536fc28c-1090-42b2-bf25-9970898be417" containerName="maas-api" Jul 08 20:39:49.419967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419910 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e" containerName="maas-api" Jul 08 20:39:49.419967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419920 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e" containerName="maas-api" Jul 08 20:39:49.419967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419933 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" Jul 08 20:39:49.419967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419941 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" Jul 08 20:39:49.419967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419952 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" Jul 08 20:39:49.419967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419960 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" Jul 08 20:39:49.420293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419976 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" Jul 08 20:39:49.420293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.419984 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" Jul 08 20:39:49.420293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.420009 2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" Jul 08 20:39:49.420293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.420018 2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" Jul 08 20:39:49.420293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.420112 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa7edf6-85fb-4fe8-bd34-9d0d41407d8e" containerName="maas-api" Jul 08 20:39:49.420293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.420124 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" Jul 08 20:39:49.420293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.420132 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" Jul 08 20:39:49.420293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.420141 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="de7e81f8-2934-4a71-a12c-2ad946dcf626" containerName="istio-proxy" Jul 08 20:39:49.420293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.420156 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="536fc28c-1090-42b2-bf25-9970898be417" containerName="maas-api" Jul 08 20:39:49.420734 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.420358 2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a73a27f9-33a1-493e-9ca0-0af5bb8d5cb1" containerName="istio-proxy" Jul 08 20:39:49.422425 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.422398 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:49.422874 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.422860 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gq7f/must-gather-nrdlq" Jul 08 20:39:49.425425 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.425406 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gq7f\"/\"openshift-service-ca.crt\"" Jul 08 20:39:49.426458 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.426442 2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9gq7f\"/\"kube-root-ca.crt\"" Jul 08 20:39:49.426563 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.426453 2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9gq7f\"/\"default-dockercfg-bbkdr\"" Jul 08 20:39:49.430755 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.430736 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gq7f/must-gather-nrdlq"] Jul 08 20:39:49.478967 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.478935 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:39:49.524733 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.524705 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f-must-gather-output\") pod \"must-gather-nrdlq\" (UID: \"38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f\") " pod="openshift-must-gather-9gq7f/must-gather-nrdlq" Jul 08 20:39:49.524922 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.524755 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vts9g\" (UniqueName: \"kubernetes.io/projected/38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f-kube-api-access-vts9g\") pod \"must-gather-nrdlq\" (UID: \"38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f\") " pod="openshift-must-gather-9gq7f/must-gather-nrdlq" Jul 08 20:39:49.626081 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.626046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f-must-gather-output\") pod \"must-gather-nrdlq\" (UID: \"38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f\") " pod="openshift-must-gather-9gq7f/must-gather-nrdlq" Jul 08 20:39:49.626262 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.626100 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts9g\" (UniqueName: \"kubernetes.io/projected/38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f-kube-api-access-vts9g\") pod \"must-gather-nrdlq\" (UID: \"38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f\") " pod="openshift-must-gather-9gq7f/must-gather-nrdlq" Jul 08 20:39:49.626432 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.626414 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f-must-gather-output\") pod \"must-gather-nrdlq\" (UID: \"38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f\") " pod="openshift-must-gather-9gq7f/must-gather-nrdlq" Jul 08 20:39:49.634176 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.634142 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vts9g\" (UniqueName: \"kubernetes.io/projected/38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f-kube-api-access-vts9g\") pod \"must-gather-nrdlq\" (UID: \"38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f\") " pod="openshift-must-gather-9gq7f/must-gather-nrdlq" Jul 08 20:39:49.732491 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.732405 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gq7f/must-gather-nrdlq" Jul 08 20:39:49.865262 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.865125 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gq7f/must-gather-nrdlq"] Jul 08 20:39:49.867395 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:39:49.867369 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38cef7f0_f5b4_40d5_97eb_a12a7ac3eb7f.slice/crio-3cb0107ddf439200b57d72010a973bb4100a8d7c9c73a2a7bada113166c48446 WatchSource:0}: Error finding container 3cb0107ddf439200b57d72010a973bb4100a8d7c9c73a2a7bada113166c48446: Status 404 returned error can't find the container with id 3cb0107ddf439200b57d72010a973bb4100a8d7c9c73a2a7bada113166c48446 Jul 08 20:39:49.869269 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:49.869256 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 20:39:50.528293 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:50.528258 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gq7f/must-gather-nrdlq" event={"ID":"38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f","Type":"ContainerStarted","Data":"3cb0107ddf439200b57d72010a973bb4100a8d7c9c73a2a7bada113166c48446"} Jul 08 20:39:51.535567 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:51.535515 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gq7f/must-gather-nrdlq" event={"ID":"38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f","Type":"ContainerStarted","Data":"4c4b352f4b8d03ddb9f43d5ccc5f0787d7e0a3847630145ef0bdb8f058a67a72"} Jul 08 20:39:51.535953 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:51.535580 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gq7f/must-gather-nrdlq" event={"ID":"38cef7f0-f5b4-40d5-97eb-a12a7ac3eb7f","Type":"ContainerStarted","Data":"da4035d0c7a3694b781ca3ed27c01e604ac8468ac8a9e5787195dd1cc3d778a6"} Jul 08 20:39:51.551442 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:51.551362 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9gq7f/must-gather-nrdlq" podStartSLOduration=1.387537378 podStartE2EDuration="2.551336524s" totalImagesPullingTime="1.163799146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 20:39:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:39:51.54928074 +0000 UTC m=+2359.546297676" watchObservedRunningTime="2026-07-08 20:39:51.551336524 +0000 UTC m=+2359.548353463" Jul 08 20:39:52.556545 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:52.556504 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4jt2s_938c8e6d-1bc6-432e-a7fd-f1c3da1bf189/global-pull-secret-syncer/0.log" Jul 08 20:39:52.678662 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:52.678630 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" Jul 08 20:39:52.771556 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:52.771518 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-18.ec2.internal_811aeb70eccf3bdd8a847e099989ebe9/haproxy/0.log" Jul 08 20:39:57.698552 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:57.698492 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-p4x4k_fd398bc2-4cc4-4d1f-801f-08a7aa0322ca/manager/0.log" Jul 08 20:39:57.728402 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:57.728367 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-pvzpn_7a4b6b9a-d0e7-4f22-9ded-94e70ed19098/kuadrant-console-plugin/0.log" Jul 08 20:39:59.735024 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.734993 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31b84820-0600-44c6-971a-1815fd067ed2/alertmanager/0.log" Jul 08 20:39:59.758281 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.758221 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31b84820-0600-44c6-971a-1815fd067ed2/config-reloader/0.log" Jul 08 20:39:59.780582 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.780539 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31b84820-0600-44c6-971a-1815fd067ed2/kube-rbac-proxy-web/0.log" Jul 08 20:39:59.803070 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.803043 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31b84820-0600-44c6-971a-1815fd067ed2/kube-rbac-proxy/0.log" Jul 08 20:39:59.828279 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.828244 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31b84820-0600-44c6-971a-1815fd067ed2/kube-rbac-proxy-metric/0.log" Jul 08 20:39:59.852982 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.852951 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31b84820-0600-44c6-971a-1815fd067ed2/prom-label-proxy/0.log" Jul 08 20:39:59.875555 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.875523 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_31b84820-0600-44c6-971a-1815fd067ed2/init-config-reloader/0.log" Jul 08 20:39:59.942183 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.942150 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-prtvx_ebd1b6a4-141b-4e6c-9727-56fc32ed1660/kube-state-metrics/0.log" Jul 08 20:39:59.960091 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.960058 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-prtvx_ebd1b6a4-141b-4e6c-9727-56fc32ed1660/kube-rbac-proxy-main/0.log" Jul 08 20:39:59.993804 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:39:59.993770 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-prtvx_ebd1b6a4-141b-4e6c-9727-56fc32ed1660/kube-rbac-proxy-self/0.log" Jul 08 20:40:00.029111 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.029073 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-c6cfb6b5c-srf65_3ba3c834-b8c6-4574-8f48-cbbeb086589f/metrics-server/0.log" Jul 08 20:40:00.056974 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.056941 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-m67n4_ed5f81ba-c6f4-42ed-aeb7-46e910fd9406/monitoring-plugin/0.log" Jul 08 20:40:00.089262 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.089230 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lk7v_2f6a1c34-f16e-4352-9877-ee72b6b71fa7/node-exporter/0.log" Jul 08 20:40:00.110509 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.110477 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lk7v_2f6a1c34-f16e-4352-9877-ee72b6b71fa7/kube-rbac-proxy/0.log" Jul 08 20:40:00.133228 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.133197 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6lk7v_2f6a1c34-f16e-4352-9877-ee72b6b71fa7/init-textfile/0.log" Jul 08 20:40:00.360589 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.360558 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-qnfzn_629383ec-cbbe-4daf-bf3f-d13ed29b5440/kube-rbac-proxy-main/0.log" Jul 08 20:40:00.386935 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.386903 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-qnfzn_629383ec-cbbe-4daf-bf3f-d13ed29b5440/kube-rbac-proxy-self/0.log" Jul 08 20:40:00.409113 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.409061 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-qnfzn_629383ec-cbbe-4daf-bf3f-d13ed29b5440/openshift-state-metrics/0.log" Jul 08 20:40:00.447201 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.447168 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17990734-0dd8-492b-8df8-f63a629f59e2/prometheus/0.log" Jul 08 20:40:00.469326 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.469283 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17990734-0dd8-492b-8df8-f63a629f59e2/config-reloader/0.log" Jul 08 20:40:00.492925 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.492897 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17990734-0dd8-492b-8df8-f63a629f59e2/thanos-sidecar/0.log" Jul 08 20:40:00.515739 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.515662 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17990734-0dd8-492b-8df8-f63a629f59e2/kube-rbac-proxy-web/0.log" Jul 08 20:40:00.539577 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.539554 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17990734-0dd8-492b-8df8-f63a629f59e2/kube-rbac-proxy/0.log" Jul 08 20:40:00.561603 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.561574 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17990734-0dd8-492b-8df8-f63a629f59e2/kube-rbac-proxy-thanos/0.log" Jul 08 20:40:00.584807 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.584782 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_17990734-0dd8-492b-8df8-f63a629f59e2/init-config-reloader/0.log" Jul 08 20:40:00.623652 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.623625 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-w4fn9_61819ca7-d0ea-4877-805b-d237f793ddd7/prometheus-operator/0.log" Jul 08 20:40:00.644251 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.644223 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-w4fn9_61819ca7-d0ea-4877-805b-d237f793ddd7/kube-rbac-proxy/0.log" Jul 08 20:40:00.668301 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.668274 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-ffngq_3fcb04b1-40e5-401d-aecf-8304c3b4933f/prometheus-operator-admission-webhook/0.log" Jul 08 20:40:00.699884 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.699823 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bbf4dc7b9-frb7p_5407d5d5-b968-4216-ab1b-25dd81905065/telemeter-client/0.log" Jul 08 20:40:00.727842 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.727812 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bbf4dc7b9-frb7p_5407d5d5-b968-4216-ab1b-25dd81905065/reload/0.log" Jul 08 20:40:00.751050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.751016 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bbf4dc7b9-frb7p_5407d5d5-b968-4216-ab1b-25dd81905065/kube-rbac-proxy/0.log" Jul 08 20:40:00.801982 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.801955 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68655c9874-zf74w_fe56b542-818b-468b-9d93-b8c12d6b0b9a/thanos-query/0.log" Jul 08 20:40:00.823514 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.823487 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68655c9874-zf74w_fe56b542-818b-468b-9d93-b8c12d6b0b9a/kube-rbac-proxy-web/0.log" Jul 08 20:40:00.847370 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.847336 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68655c9874-zf74w_fe56b542-818b-468b-9d93-b8c12d6b0b9a/kube-rbac-proxy/0.log" Jul 08 20:40:00.870645 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.870618 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68655c9874-zf74w_fe56b542-818b-468b-9d93-b8c12d6b0b9a/prom-label-proxy/0.log" Jul 08 20:40:00.893188 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.893157 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68655c9874-zf74w_fe56b542-818b-468b-9d93-b8c12d6b0b9a/kube-rbac-proxy-rules/0.log" Jul 08 20:40:00.913285 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:00.913255 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-68655c9874-zf74w_fe56b542-818b-468b-9d93-b8c12d6b0b9a/kube-rbac-proxy-metrics/0.log" Jul 08 20:40:03.138135 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.138103 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67545bd57d-fmk76_36af0242-cb8c-483d-a569-2bd951759e98/console/0.log" Jul 08 20:40:03.196906 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.196862 2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5"] Jul 08 20:40:03.205842 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.205814 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.209509 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.209471 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5"] Jul 08 20:40:03.278048 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.278016 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-sys\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.278264 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.278058 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-podres\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.278264 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.278124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-lib-modules\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.278264 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.278153 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckbkh\" (UniqueName: \"kubernetes.io/projected/e0ca23d8-bf03-4182-baf9-b9945c5e8408-kube-api-access-ckbkh\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.278264 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.278199 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-proc\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.378859 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.378823 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckbkh\" (UniqueName: \"kubernetes.io/projected/e0ca23d8-bf03-4182-baf9-b9945c5e8408-kube-api-access-ckbkh\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.379050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.378904 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-proc\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.379050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.378961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-sys\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.379050 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.378995 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-podres\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.379207 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.379057 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-lib-modules\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.379207 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.379055 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-proc\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.379207 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.379108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-sys\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.379207 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.379161 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-podres\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.379207 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.379190 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0ca23d8-bf03-4182-baf9-b9945c5e8408-lib-modules\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.386694 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.386669 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckbkh\" (UniqueName: \"kubernetes.io/projected/e0ca23d8-bf03-4182-baf9-b9945c5e8408-kube-api-access-ckbkh\") pod \"perf-node-gather-daemonset-zwqr5\" (UID: \"e0ca23d8-bf03-4182-baf9-b9945c5e8408\") " pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.522331 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.522269 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:03.698750 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:03.698708 2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5"] Jul 08 20:40:03.703111 ip-10-0-132-18 kubenswrapper[2573]: W0708 20:40:03.703067 2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode0ca23d8_bf03_4182_baf9_b9945c5e8408.slice/crio-1adc3d5206a11c6a762ae2add4e68bc75d43e2d4ddb8085143a2a8d1eac9391e WatchSource:0}: Error finding container 1adc3d5206a11c6a762ae2add4e68bc75d43e2d4ddb8085143a2a8d1eac9391e: Status 404 returned error can't find the container with id 1adc3d5206a11c6a762ae2add4e68bc75d43e2d4ddb8085143a2a8d1eac9391e Jul 08 20:40:04.634857 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:04.634826 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" event={"ID":"e0ca23d8-bf03-4182-baf9-b9945c5e8408","Type":"ContainerStarted","Data":"82d93bb457f2618e83347f4581ed9f00a1b8dc8c5b54cf855c5327028e2fedfd"} Jul 08 20:40:04.635252 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:04.634876 2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" event={"ID":"e0ca23d8-bf03-4182-baf9-b9945c5e8408","Type":"ContainerStarted","Data":"1adc3d5206a11c6a762ae2add4e68bc75d43e2d4ddb8085143a2a8d1eac9391e"} Jul 08 20:40:04.635252 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:04.634946 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:04.914874 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:04.914791 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v7nfw_f4cb57bc-18c6-4bde-846c-d3a761bb4848/dns/0.log" Jul 08 20:40:04.936203 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:04.936176 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v7nfw_f4cb57bc-18c6-4bde-846c-d3a761bb4848/kube-rbac-proxy/0.log" Jul 08 20:40:05.053837 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:05.053806 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-w5dc5_a35b692c-4d5f-4e7a-addd-2ed017343ab4/dns-node-resolver/0.log" Jul 08 20:40:05.614749 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:05.614716 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5tkjn_f05057eb-dc4d-45bd-adf9-1e20fdb9c027/node-ca/0.log" Jul 08 20:40:06.731241 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:06.731213 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789db8bf69-ftszt_df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597/kube-auth-proxy/0.log" Jul 08 20:40:06.833124 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:06.833094 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-d9fpz_fa609f0b-a36a-45ea-a12c-4ad343179ff7/payload-pre-processing/0.log" Jul 08 20:40:07.548664 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:07.548638 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cwvxn_611e0c1c-93b6-4a90-8165-7128c8ff6be9/serve-healthcheck-canary/0.log" Jul 08 20:40:08.090419 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:08.090391 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lmwhx_d128df57-bf08-4aeb-9513-d64f27bcf180/kube-rbac-proxy/0.log" Jul 08 20:40:08.112374 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:08.112342 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lmwhx_d128df57-bf08-4aeb-9513-d64f27bcf180/exporter/0.log" Jul 08 20:40:08.135094 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:08.135063 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lmwhx_d128df57-bf08-4aeb-9513-d64f27bcf180/extractor/0.log" Jul 08 20:40:10.653343 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:10.653302 2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" Jul 08 20:40:10.669268 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:10.669206 2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9gq7f/perf-node-gather-daemonset-zwqr5" podStartSLOduration=7.669185447 podStartE2EDuration="7.669185447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 20:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 20:40:04.652090578 +0000 UTC m=+2372.649107515" watchObservedRunningTime="2026-07-08 20:40:10.669185447 +0000 UTC m=+2378.666202384" Jul 08 20:40:10.803354 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:10.803326 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67959458dc-fdjcc_d9ecda20-8582-4251-83cb-706e1b13f347/manager/0.log" Jul 08 20:40:10.824721 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:10.824696 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-67959458dc-fdjcc_d9ecda20-8582-4251-83cb-706e1b13f347/copy-manifests/0.log" Jul 08 20:40:12.124654 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:12.124522 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-557dbd68d4-4g9g8_6ef44d1d-7d7e-4346-8c21-83d38fff475c/manager/0.log" Jul 08 20:40:12.181344 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:12.181250 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-79bfv_865977f3-fd8b-45f3-867c-951902a420ea/openshift-lws-operator/0.log" Jul 08 20:40:13.151830 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:13.151792 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789db8bf69-ftszt_df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597/kube-auth-proxy/0.log" Jul 08 20:40:13.242422 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:13.241082 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-d9fpz_fa609f0b-a36a-45ea-a12c-4ad343179ff7/payload-pre-processing/0.log" Jul 08 20:40:14.532773 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:14.532740 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-789db8bf69-ftszt_df5d7fb6-8f6b-4bba-bbf9-b69f7e76e597/kube-auth-proxy/0.log" Jul 08 20:40:14.619720 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:14.619689 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-d9fpz_fa609f0b-a36a-45ea-a12c-4ad343179ff7/payload-pre-processing/0.log" Jul 08 20:40:15.253775 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:15.253742 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:40:15.358620 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:15.358538 2573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-54tqn_ba630e65-9372-469b-8b3e-9d88d405720e/konnectivity-agent/0.log" limit=1024 Jul 08 20:40:19.119268 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.119236 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjhks_33019580-9b46-4bda-a0dd-3687d6e3026b/kube-multus-additional-cni-plugins/0.log" Jul 08 20:40:19.143868 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.143841 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjhks_33019580-9b46-4bda-a0dd-3687d6e3026b/egress-router-binary-copy/0.log" Jul 08 20:40:19.168111 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.168083 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjhks_33019580-9b46-4bda-a0dd-3687d6e3026b/cni-plugins/0.log" Jul 08 20:40:19.189559 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.189535 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjhks_33019580-9b46-4bda-a0dd-3687d6e3026b/bond-cni-plugin/0.log" Jul 08 20:40:19.211797 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.211760 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjhks_33019580-9b46-4bda-a0dd-3687d6e3026b/routeoverride-cni/0.log" Jul 08 20:40:19.233297 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.233272 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjhks_33019580-9b46-4bda-a0dd-3687d6e3026b/whereabouts-cni-bincopy/0.log" Jul 08 20:40:19.259722 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.259695 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zjhks_33019580-9b46-4bda-a0dd-3687d6e3026b/whereabouts-cni/0.log" Jul 08 20:40:19.324788 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.324756 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-sz827_bd247e9d-54a0-4db5-a1a7-404353527533/kube-multus/0.log" Jul 08 20:40:19.393792 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.393714 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5glh6_2cb1ff8c-6dc6-4981-a487-b73961dbd578/network-metrics-daemon/0.log" Jul 08 20:40:19.412585 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:19.412556 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5glh6_2cb1ff8c-6dc6-4981-a487-b73961dbd578/kube-rbac-proxy/0.log" Jul 08 20:40:20.500298 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:20.500271 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-controller/0.log" Jul 08 20:40:20.517134 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:20.517105 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/1.log" Jul 08 20:40:20.517134 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:20.517135 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovn-acl-logging/0.log" Jul 08 20:40:20.534095 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:20.534068 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/kube-rbac-proxy-node/0.log" Jul 08 20:40:20.558913 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:20.558884 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/kube-rbac-proxy-ovn-metrics/0.log" Jul 08 20:40:20.580769 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:20.580745 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/northd/0.log" Jul 08 20:40:20.602431 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:20.602402 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/nbdb/0.log" Jul 08 20:40:20.625874 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:20.625844 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/sbdb/0.log" Jul 08 20:40:20.690301 ip-10-0-132-18 kubenswrapper[2573]: I0708 20:40:20.690270 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdwjg_390ebffc-4753-4b01-9f86-8116898f4c36/ovnkube-controller/0.log"