Jul 24 15:33:55.257303 ip-10-0-132-156 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 15:33:55.257317 ip-10-0-132-156 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 15:33:55.257326 ip-10-0-132-156 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 15:33:55.257703 ip-10-0-132-156 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 15:34:05.262501 ip-10-0-132-156 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 15:34:05.262513 ip-10-0-132-156 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 46da42318afb48aa8efa95a6eea04e49 -- Jul 24 15:35:57.524995 ip-10-0-132-156 systemd[1]: Starting Kubernetes Kubelet... Jul 24 15:35:57.892355 ip-10-0-132-156 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:35:57.892355 ip-10-0-132-156 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 15:35:57.892355 ip-10-0-132-156 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:35:57.892355 ip-10-0-132-156 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 15:35:57.892355 ip-10-0-132-156 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:35:57.893998 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.893909 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 15:35:57.898654 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898638 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:35:57.898654 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898655 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898661 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898664 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898667 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898670 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898673 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898677 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898680 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898683 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898697 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898700 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898703 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898705 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898708 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898711 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898714 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898716 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898719 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898723 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898726 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:35:57.898733 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898729 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898731 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898735 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898738 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898740 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898743 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898746 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898749 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898752 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898754 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898757 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898760 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898762 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898764 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898767 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898770 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898772 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898775 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898777 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898780 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:35:57.899245 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898783 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898785 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898788 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898790 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898792 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898795 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898797 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898800 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898804 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898807 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898810 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898816 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898819 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898821 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898825 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898827 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898830 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898833 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898835 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:35:57.899758 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898838 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898840 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898843 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898846 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898848 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898851 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898853 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898856 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898858 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898861 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898863 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898866 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898868 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898870 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898873 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898875 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898879 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898881 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898885 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:35:57.900221 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898887 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898890 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898892 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898895 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898897 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898900 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898902 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898905 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898907 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898910 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898913 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898915 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898917 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898920 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898922 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898925 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898928 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898930 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898933 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:35:57.900670 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.898935 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:35:57.901806 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901793 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:35:57.901806 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901806 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901810 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901813 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901816 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901818 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901822 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901826 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901829 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901832 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901835 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901837 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901840 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901843 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901846 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901848 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901851 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901854 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901856 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901860 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901862 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:35:57.901868 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901865 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901867 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901870 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901873 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901875 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901878 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901881 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901883 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901886 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901888 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901891 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901893 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901896 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901898 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901901 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901903 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901906 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901908 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901911 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901913 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:35:57.902368 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901916 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901918 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901920 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901922 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901925 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901927 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901929 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901932 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901935 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901937 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901941 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901945 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901948 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901951 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901954 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901957 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901960 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901962 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901964 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:35:57.902880 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901967 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901969 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901972 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901974 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901977 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901979 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901981 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901984 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901987 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901989 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901991 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901994 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901996 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.901999 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902001 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902004 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902007 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902009 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902012 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902016 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:35:57.903353 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902019 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902021 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902024 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902026 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902028 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902032 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902034 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902037 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902040 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902042 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902045 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902048 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902050 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902053 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902056 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902058 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902061 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902063 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902066 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902424 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:35:57.903862 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902429 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902433 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902435 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902438 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902440 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902443 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902446 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902448 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902450 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902453 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902456 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902459 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902461 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902463 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902466 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902468 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902470 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902473 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902476 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902479 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902481 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:35:57.904358 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902484 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902486 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902488 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902491 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902493 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902496 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902498 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902500 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902503 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902505 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902508 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902511 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902513 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902515 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902518 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902520 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902523 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902525 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902528 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:35:57.904929 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902530 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902532 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902535 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902537 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902540 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902542 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902544 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902547 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902550 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902553 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902556 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902558 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902561 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902564 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902566 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902568 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902571 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902573 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902575 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902578 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:35:57.905400 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902580 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902582 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902585 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902587 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902590 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902592 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902596 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902599 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902603 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902606 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902609 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902612 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902615 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902617 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902621 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902624 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902627 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902631 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902633 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:35:57.905905 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902636 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902638 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902641 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902644 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902647 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902650 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902653 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902655 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902658 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902660 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902662 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902665 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902667 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902669 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902672 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902674 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902677 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902679 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.902681 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902773 2571 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 15:35:57.906605 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902783 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902792 2571 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902799 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902805 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902809 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902813 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902817 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902823 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902826 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902829 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902832 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902835 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902839 2571 flags.go:64] FLAG: --cgroup-root="" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902841 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902844 2571 flags.go:64] FLAG: --client-ca-file="" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902847 2571 flags.go:64] FLAG: --cloud-provider="external" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902850 2571 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902854 2571 flags.go:64] FLAG: --cluster-domain="" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902856 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902860 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902863 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902866 2571 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902869 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 15:35:57.907103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902872 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902875 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902878 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902881 2571 flags.go:64] FLAG: --contention-profiling="false" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902884 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902887 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902890 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902893 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902897 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902900 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902903 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902908 2571 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902911 2571 flags.go:64] FLAG: --enable-server="true" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902914 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902918 2571 flags.go:64] FLAG: --event-burst="100" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902921 2571 flags.go:64] FLAG: --event-qps="50" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902925 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902928 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902931 2571 flags.go:64] FLAG: --eviction-hard="" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902934 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902937 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902941 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902944 2571 flags.go:64] FLAG: --eviction-soft="" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902947 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902949 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 15:35:57.907657 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902952 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902955 2571 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902958 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902961 2571 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902964 2571 flags.go:64] FLAG: --feature-gates="" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902967 2571 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902970 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902973 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902976 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902979 2571 flags.go:64] FLAG: --healthz-port="10248" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902981 2571 flags.go:64] FLAG: --help="false" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902984 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-132-156.ec2.internal" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902987 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902989 2571 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902992 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902995 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.902999 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903001 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903004 2571 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903006 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903009 2571 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903012 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903014 2571 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903018 2571 flags.go:64] FLAG: --kube-reserved="" Jul 24 15:35:57.908371 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903021 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903023 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903027 2571 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903029 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903033 2571 flags.go:64] FLAG: --lock-file="" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903036 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903038 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903041 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903047 2571 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903049 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903052 2571 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903055 2571 flags.go:64] FLAG: --logging-format="text" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903058 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903061 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903064 2571 flags.go:64] FLAG: --manifest-url="" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903066 2571 flags.go:64] FLAG: --manifest-url-header="" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903070 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903073 2571 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903077 2571 flags.go:64] FLAG: --max-pods="110" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903080 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903082 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903085 2571 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903088 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903092 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903094 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 15:35:57.908973 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903097 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903103 2571 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903106 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903109 2571 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903112 2571 flags.go:64] FLAG: --pod-cidr="" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903115 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903121 2571 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903124 2571 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903127 2571 flags.go:64] FLAG: --pods-per-core="0" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903129 2571 flags.go:64] FLAG: --port="10250" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903132 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903135 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0741cb2f164f2ebc9" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903139 2571 flags.go:64] FLAG: --qos-reserved="" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903142 2571 flags.go:64] FLAG: --read-only-port="10255" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903145 2571 flags.go:64] FLAG: --register-node="true" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903148 2571 flags.go:64] FLAG: --register-with-taints="" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903151 2571 flags.go:64] FLAG: --registry-burst="10" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903154 2571 flags.go:64] FLAG: --registry-qps="5" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903156 2571 flags.go:64] FLAG: --reserved-cpus="" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903160 2571 flags.go:64] FLAG: --reserved-memory="" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903163 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903166 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903169 2571 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903172 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903174 2571 flags.go:64] FLAG: --runonce="false" Jul 24 15:35:57.909593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903177 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903180 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903183 2571 flags.go:64] FLAG: --seccomp-default="false" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903186 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903189 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903192 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903194 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903197 2571 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903200 2571 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903203 2571 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903205 2571 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903208 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903211 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903215 2571 flags.go:64] FLAG: --system-cgroups="" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903218 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903223 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903226 2571 flags.go:64] FLAG: --tls-cert-file="" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903228 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903231 2571 flags.go:64] FLAG: --tls-min-version="" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903235 2571 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903238 2571 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903241 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903243 2571 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903246 2571 flags.go:64] FLAG: --v="2" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903250 2571 flags.go:64] FLAG: --version="false" Jul 24 15:35:57.910528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903254 2571 flags.go:64] FLAG: --vmodule="" Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903257 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.903261 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903381 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903385 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903388 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903391 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903394 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903396 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903399 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903402 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903412 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903415 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903418 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903420 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903423 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903426 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903428 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903431 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903434 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:35:57.911230 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903438 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903440 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903443 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903445 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903448 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903450 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903455 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903457 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903459 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903462 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903465 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903467 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903472 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903475 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903478 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903482 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903484 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903487 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903490 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903492 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:35:57.911753 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903495 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903498 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903500 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903503 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903506 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903508 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903511 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903513 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903515 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903518 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903520 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903523 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903528 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903531 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903534 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903537 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903539 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903542 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903545 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:35:57.912234 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903548 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903551 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903553 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903556 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903559 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903561 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903563 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903566 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903568 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903571 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903574 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903576 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903579 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903581 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903584 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903586 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903588 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903591 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903593 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:35:57.912717 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903596 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903598 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903600 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903603 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903605 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903607 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903611 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903613 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903616 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903618 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903621 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903623 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903627 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903630 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903632 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903635 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903639 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903642 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903644 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:35:57.913173 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903647 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903649 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903652 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903654 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.903657 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.904171 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.912059 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.912075 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.912094 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.912100 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912160 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912165 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912168 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912171 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:35:57.913625 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912174 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912177 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912180 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912183 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912185 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912188 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912191 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912194 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912196 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912199 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912202 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912204 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912207 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912210 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912212 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912214 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912217 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912219 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912221 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912224 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:35:57.913998 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912226 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912229 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912231 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912234 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912247 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912251 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912254 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912257 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912260 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912263 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912265 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912268 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912271 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912273 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912275 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912278 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912280 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912283 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912285 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:35:57.914472 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912288 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912290 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912293 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912296 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912299 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912301 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912303 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912306 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912308 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912311 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912313 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912315 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912318 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912320 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912322 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912325 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912328 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912331 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912334 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912336 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:35:57.914995 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912339 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912341 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912344 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912346 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912349 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912351 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912353 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912356 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912358 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912361 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912363 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912367 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912371 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912374 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912376 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912379 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912383 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912387 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912389 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:35:57.915501 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912392 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912395 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912398 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912400 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912403 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912406 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912410 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912413 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912415 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912418 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912421 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912424 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912426 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912429 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912431 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912433 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:35:57.915991 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912436 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.912441 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912554 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912559 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912562 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912565 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912567 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912570 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912573 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912575 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912578 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912580 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912583 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912585 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912589 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:35:57.916370 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912591 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912594 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912596 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912598 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912601 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912603 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912605 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912608 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912611 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912613 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912616 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912619 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912621 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912623 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912626 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912628 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912632 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912636 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912638 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912641 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:35:57.916766 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912643 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912646 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912648 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912650 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912653 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912655 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912658 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912660 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912662 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912665 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912667 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912670 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912673 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912676 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912678 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912680 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912683 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912701 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912705 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:35:57.917242 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912708 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912710 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912713 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912716 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912720 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912722 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912726 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912729 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912732 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912735 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912738 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912741 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912743 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912746 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912748 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912751 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912753 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912755 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912758 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:35:57.917747 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912760 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912763 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912765 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912768 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912770 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912772 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912775 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912778 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912781 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912783 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912786 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912789 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912791 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912794 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912796 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912799 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912801 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912804 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912806 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912809 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:35:57.918219 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912811 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912813 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912816 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912818 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912821 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912823 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912826 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:35:57.912828 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.912833 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.912939 2571 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.914905 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.916131 2571 server.go:1019] "Starting client certificate rotation" Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.916230 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:35:57.918737 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.916296 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:35:57.919055 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.918380 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:35:57.919530 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.919513 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:35:57.930339 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.930320 2571 log.go:25] "Validated CRI v1 runtime API" Jul 24 15:35:57.934030 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.934014 2571 log.go:25] "Validated CRI v1 image API" Jul 24 15:35:57.934913 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.934898 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 15:35:57.938269 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.938252 2571 fs.go:135] Filesystem UUIDs: map[56882e6b-9c2c-457f-9bee-14efebde1ffc:/dev/nvme0n1p4 77058b83-f69d-47c1-a2be-fea2f0e92a7e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 24 15:35:57.938317 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.938269 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 15:35:57.945352 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.945238 2571 manager.go:216] Machine: {Timestamp:2026-07-24 15:35:57.944118073 +0000 UTC m=+0.309851658 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102285 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec263b7af85ac8c01327f5d92df965f8 SystemUUID:ec263b7a-f85a-c8c0-1327-f5d92df965f8 BootID:46da4231-8afb-48aa-8efa-95a6eea04e49 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c1:1c:8c:09:f7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c1:1c:8c:09:f7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:63:20:7e:3e:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 15:35:57.945352 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.945342 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 15:35:57.945530 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.945447 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 15:35:57.946781 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.946764 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:35:57.947734 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.947702 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 15:35:57.947864 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.947737 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 15:35:57.948374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.948364 2571 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 15:35:57.948404 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.948376 2571 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 15:35:57.948404 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.948383 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:35:57.949085 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.949074 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:35:57.949764 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.949754 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 15:35:57.951474 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.951463 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:35:57.951586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.951577 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 15:35:57.951616 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.951606 2571 kubelet.go:486] "Attempting to sync node with API server" Jul 24 15:35:57.951616 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.951615 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 15:35:57.951666 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.951632 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 15:35:57.951666 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.951641 2571 kubelet.go:398] "Adding apiserver pod source" Jul 24 15:35:57.951666 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.951650 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 15:35:57.952007 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.951996 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:35:57.952050 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.952015 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:35:57.954294 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.954279 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 15:35:57.955919 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.955906 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 15:35:57.955972 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.955937 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 15:35:57.957270 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957259 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 15:35:57.957309 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957275 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 15:35:57.957309 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957281 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 15:35:57.957309 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957286 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 15:35:57.957309 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957296 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 15:35:57.957309 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957303 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 15:35:57.957433 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957311 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 15:35:57.957433 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957317 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 15:35:57.957433 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957324 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 15:35:57.957433 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957331 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 15:35:57.957433 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957342 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 15:35:57.957433 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.957347 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 15:35:57.958208 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.958195 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 15:35:57.958242 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.958210 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 15:35:57.961481 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.961467 2571 server.go:1268] "Started kubelet" Jul 24 15:35:57.961599 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.961546 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 15:35:57.961656 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.961627 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 15:35:57.962097 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.961581 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 15:35:57.962474 ip-10-0-132-156 systemd[1]: Started Kubernetes Kubelet. Jul 24 15:35:57.963454 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.963438 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 15:35:57.964638 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.964619 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 15:35:57.964750 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:57.964630 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 15:35:57.964750 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:57.964709 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 15:35:57.964882 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.964868 2571 server.go:310] "Adding debug handlers to kubelet server" Jul 24 15:35:57.967220 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:57.967196 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 15:35:57.969577 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.969563 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 15:35:57.969653 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.969581 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 15:35:57.969734 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:57.969669 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:35:57.969987 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:57.969188 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-156.ec2.internal.18c543c90445bc31 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-156.ec2.internal,UID:ip-10-0-132-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-156.ec2.internal,},FirstTimestamp:2026-07-24 15:35:57.961436209 +0000 UTC m=+0.327169794,LastTimestamp:2026-07-24 15:35:57.961436209 +0000 UTC m=+0.327169794,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-156.ec2.internal,}" Jul 24 15:35:57.970234 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970221 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 15:35:57.970303 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970221 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 15:35:57.970303 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970260 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 15:35:57.970408 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970376 2571 reconstruct.go:97] "Volume reconstruction finished" Jul 24 15:35:57.970408 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970384 2571 reconciler.go:29] "Reconciler: start to sync state" Jul 24 15:35:57.970539 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970468 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 15:35:57.970611 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970594 2571 factory.go:55] Registering systemd factory Jul 24 15:35:57.970655 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970614 2571 factory.go:223] Registration of the systemd container factory successfully Jul 24 15:35:57.970830 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970812 2571 factory.go:153] Registering CRI-O factory Jul 24 15:35:57.970930 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970835 2571 factory.go:223] Registration of the crio container factory successfully Jul 24 15:35:57.970930 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970857 2571 factory.go:103] Registering Raw factory Jul 24 15:35:57.970930 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.970877 2571 manager.go:1196] Started watching for new ooms in manager Jul 24 15:35:57.971207 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.971197 2571 manager.go:349] Starting recovery of all containers Jul 24 15:35:57.981211 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:57.978641 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 15:35:57.981211 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:57.978704 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 15:35:57.983750 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.983578 2571 manager.go:354] Recovery completed Jul 24 15:35:57.985345 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:57.985325 2571 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 24 15:35:57.988392 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.988380 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:35:57.990217 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.990200 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:35:57.990264 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.990235 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:35:57.990264 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.990245 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:35:57.990798 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.990785 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 15:35:57.990798 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.990797 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 15:35:57.990872 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.990803 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:35:57.990872 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.990812 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:35:57.991498 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.991484 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:35:57.991546 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.991516 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:35:57.991546 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.991527 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:35:57.992214 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.992199 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r9qf2" Jul 24 15:35:57.992452 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.992435 2571 policy_none.go:49] "None policy: Start" Jul 24 15:35:57.992452 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.992455 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 15:35:57.992567 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:57.992406 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-156.ec2.internal.18c543c905fce9d0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-156.ec2.internal,UID:ip-10-0-132-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-156.ec2.internal,},FirstTimestamp:2026-07-24 15:35:57.990218192 +0000 UTC m=+0.355951779,LastTimestamp:2026-07-24 15:35:57.990218192 +0000 UTC m=+0.355951779,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-156.ec2.internal,}" Jul 24 15:35:57.992567 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.992477 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 15:35:57.993503 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:57.993491 2571 policy_none.go:47] "Start" Jul 24 15:35:58.002430 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.002413 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r9qf2" Jul 24 15:35:58.002567 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.002505 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-156.ec2.internal.18c543c905fd3f8d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-156.ec2.internal,UID:ip-10-0-132-156.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-156.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-156.ec2.internal,},FirstTimestamp:2026-07-24 15:35:57.990240141 +0000 UTC m=+0.355973727,LastTimestamp:2026-07-24 15:35:57.990240141 +0000 UTC m=+0.355973727,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-156.ec2.internal,}" Jul 24 15:35:58.032292 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.032275 2571 manager.go:337] "Starting Device Plugin manager" Jul 24 15:35:58.032398 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.032321 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 15:35:58.032398 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.032335 2571 server.go:87] "Starting device plugin registration server" Jul 24 15:35:58.032645 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.032633 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 15:35:58.032730 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.032661 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 15:35:58.032793 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.032751 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 15:35:58.032862 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.032849 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 15:35:58.032908 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.032864 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 15:35:58.033464 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.033443 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 15:35:58.033554 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.033496 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:35:58.109707 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.109660 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 15:35:58.111065 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.111047 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 15:35:58.111130 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.111120 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 15:35:58.111169 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.111157 2571 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 15:35:58.111215 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.111197 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 15:35:58.113637 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.113611 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:35:58.133565 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.133534 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:35:58.134437 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.134421 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:35:58.134543 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.134457 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:35:58.134543 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.134483 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:35:58.134543 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.134525 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.143725 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.143647 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.143725 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.143682 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-156.ec2.internal\": node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:35:58.166844 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.166815 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:35:58.211665 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.211616 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal"] Jul 24 15:35:58.211850 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.211736 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:35:58.213322 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.213303 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:35:58.213432 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.213338 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:35:58.213432 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.213351 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:35:58.214492 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.214476 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:35:58.214612 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.214596 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.214681 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.214642 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:35:58.215239 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.215220 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:35:58.215342 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.215259 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:35:58.215342 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.215270 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:35:58.215342 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.215289 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:35:58.215342 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.215307 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:35:58.215342 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.215316 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:35:58.216891 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.216875 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.216949 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.216908 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:35:58.217604 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.217583 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:35:58.217671 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.217637 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:35:58.217671 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.217654 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:35:58.241349 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.241321 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-156.ec2.internal\" not found" node="ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.245804 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.245787 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-156.ec2.internal\" not found" node="ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.267893 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.267871 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:35:58.272249 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.272227 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3350393ac72409ced117e07620daf2a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal\" (UID: \"d3350393ac72409ced117e07620daf2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.272318 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.272257 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3350393ac72409ced117e07620daf2a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal\" (UID: \"d3350393ac72409ced117e07620daf2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.272318 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.272274 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcc72598c4b952733bb79430abb4e958-config\") pod \"kube-apiserver-proxy-ip-10-0-132-156.ec2.internal\" (UID: \"fcc72598c4b952733bb79430abb4e958\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.368460 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.368415 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:35:58.372784 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.372763 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3350393ac72409ced117e07620daf2a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal\" (UID: \"d3350393ac72409ced117e07620daf2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.372851 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.372794 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcc72598c4b952733bb79430abb4e958-config\") pod \"kube-apiserver-proxy-ip-10-0-132-156.ec2.internal\" (UID: \"fcc72598c4b952733bb79430abb4e958\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.372851 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.372811 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3350393ac72409ced117e07620daf2a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal\" (UID: \"d3350393ac72409ced117e07620daf2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.372915 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.372856 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d3350393ac72409ced117e07620daf2a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal\" (UID: \"d3350393ac72409ced117e07620daf2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.372915 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.372870 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d3350393ac72409ced117e07620daf2a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal\" (UID: \"d3350393ac72409ced117e07620daf2a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.372915 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.372894 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fcc72598c4b952733bb79430abb4e958-config\") pod \"kube-apiserver-proxy-ip-10-0-132-156.ec2.internal\" (UID: \"fcc72598c4b952733bb79430abb4e958\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.469266 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.469186 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:35:58.543617 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.543590 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.549246 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:35:58.549225 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" Jul 24 15:35:58.570075 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.570049 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:35:58.670546 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:58.670509 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:35:59.903999 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:35:59.903973 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:36:00.004538 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:00.004501 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:36:00.048826 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.048807 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 15:36:00.049298 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.048981 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 15:36:00.105147 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:00.105122 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:36:00.139352 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.139267 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 15:30:59 +0000 UTC" deadline="2027-12-21 13:02:44.350726924 +0000 UTC" Jul 24 15:36:00.139352 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.139307 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12357h26m44.211425146s" Jul 24 15:36:00.140344 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.140331 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:36:00.141372 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.141355 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:36:00.141661 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.141642 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:36:00.141751 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.141677 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:36:00.205574 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:00.205547 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-156.ec2.internal\" not found" Jul 24 15:36:00.255812 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.255791 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:36:00.268529 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.268509 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:36:00.303665 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.303639 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal" Jul 24 15:36:00.315496 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.315476 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:36:00.316898 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.316884 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" Jul 24 15:36:00.330072 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.330051 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:36:00.353966 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:00.353938 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3350393ac72409ced117e07620daf2a.slice/crio-3bbc52fec2c7c24df60403f1ee9e4b7099a82ed9aa9ffb284d41606d13bdf30c WatchSource:0}: Error finding container 3bbc52fec2c7c24df60403f1ee9e4b7099a82ed9aa9ffb284d41606d13bdf30c: Status 404 returned error can't find the container with id 3bbc52fec2c7c24df60403f1ee9e4b7099a82ed9aa9ffb284d41606d13bdf30c Jul 24 15:36:00.354413 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:00.354384 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc72598c4b952733bb79430abb4e958.slice/crio-3e838583f568786f6d403005675613e28330fd623c64974bc75741e500a802a1 WatchSource:0}: Error finding container 3e838583f568786f6d403005675613e28330fd623c64974bc75741e500a802a1: Status 404 returned error can't find the container with id 3e838583f568786f6d403005675613e28330fd623c64974bc75741e500a802a1 Jul 24 15:36:00.358310 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.358298 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:36:00.676875 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:00.676853 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:36:01.017402 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.017318 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:36:01.085714 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.085664 2571 apiserver.go:52] "Watching apiserver" Jul 24 15:36:01.094993 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.094970 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 15:36:01.095489 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.095465 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-fthc7","openshift-ovn-kubernetes/ovnkube-node-czgtt","kube-system/konnectivity-agent-mw8jp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5","openshift-cluster-node-tuning-operator/tuned-qqr6d","openshift-image-registry/node-ca-b9smv","openshift-multus/network-metrics-daemon-4hnrl","openshift-network-diagnostics/network-check-target-df4sc","kube-system/global-pull-secret-syncer-rbhv6","kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal","openshift-dns/node-resolver-pl8t9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal","openshift-multus/multus-additional-cni-plugins-s7vsx","openshift-multus/multus-dzpj8"] Jul 24 15:36:01.098179 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.098156 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:01.098276 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.098250 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:01.100206 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.100186 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.102563 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.102540 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:01.102963 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.102937 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 15:36:01.103824 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.103514 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 15:36:01.103824 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.103528 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 15:36:01.103824 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.103517 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 15:36:01.103824 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.103518 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6wnm\"" Jul 24 15:36:01.103824 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.103730 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 15:36:01.103824 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.103737 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 15:36:01.103824 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.103828 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.104774 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.104756 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.104895 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.104881 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 15:36:01.105170 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.105157 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 15:36:01.108105 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.106197 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.108105 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.106296 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wsn8d\"" Jul 24 15:36:01.108105 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.107674 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 15:36:01.108105 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.107755 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 15:36:01.108105 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.107774 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 15:36:01.108105 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.107914 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vglbh\"" Jul 24 15:36:01.108105 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.107977 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 15:36:01.108466 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.108253 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-km2d8\"" Jul 24 15:36:01.108466 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.108333 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:36:01.110095 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.109170 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 15:36:01.110095 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.109540 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b9n24\"" Jul 24 15:36:01.110095 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.109930 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 15:36:01.110263 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.109935 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 15:36:01.110815 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.110788 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.111190 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.111164 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:01.112642 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.112623 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.113133 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113114 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:01.113218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113173 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-socket-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.113218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113200 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-sys-fs\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.113405 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113222 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-cnibin\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.113405 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.113405 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113271 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.113405 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113310 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-etc-selinux\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.113405 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113342 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-system-cni-dir\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.113405 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-os-release\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.113405 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113395 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.113756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113422 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.113756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113446 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.113756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113482 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sxc4\" (UniqueName: \"kubernetes.io/projected/b83152b7-d9db-42fc-8792-f6d38f731245-kube-api-access-7sxc4\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:01.113756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113505 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/387a88ac-aa70-4b5f-8f71-8ff684c047a8-agent-certs\") pod \"konnectivity-agent-mw8jp\" (UID: \"387a88ac-aa70-4b5f-8f71-8ff684c047a8\") " pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:01.113756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113528 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/387a88ac-aa70-4b5f-8f71-8ff684c047a8-konnectivity-ca\") pod \"konnectivity-agent-mw8jp\" (UID: \"387a88ac-aa70-4b5f-8f71-8ff684c047a8\") " pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:01.113756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113557 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-registration-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.113756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-device-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.113756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113623 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th47s\" (UniqueName: \"kubernetes.io/projected/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-kube-api-access-th47s\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.113756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.113648 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7qs\" (UniqueName: \"kubernetes.io/projected/e845a4eb-a61f-4035-8dce-2a2600cd7952-kube-api-access-qr7qs\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.115163 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.115143 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8q6l9\"" Jul 24 15:36:01.115266 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.115246 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:36:01.115353 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.115340 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 15:36:01.115409 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.115382 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 15:36:01.115580 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.115564 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:36:01.116026 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.116008 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.119281 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.117935 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.119281 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.118556 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 15:36:01.119281 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.119186 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 15:36:01.119478 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.119439 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 15:36:01.119662 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.119639 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qj62g\"" Jul 24 15:36:01.119813 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.119792 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 15:36:01.119813 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.119808 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 15:36:01.120259 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.119799 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 15:36:01.120461 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.120442 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 15:36:01.120565 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.120546 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fdrpd\"" Jul 24 15:36:01.120747 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.120729 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.122179 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.122159 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:01.122268 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.122212 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:01.122907 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.122890 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 15:36:01.123344 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.123323 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l7vfx\"" Jul 24 15:36:01.123562 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.123332 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 15:36:01.141813 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.141787 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rdk4r" Jul 24 15:36:01.150008 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.149987 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rdk4r" Jul 24 15:36:01.205189 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.205159 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 15:36:01.214143 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214110 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-os-release\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.214312 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214152 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.214312 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214181 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.214312 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-slash\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.214312 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214238 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93f95dad-8a63-4554-991b-196c269595e8-iptables-alerter-script\") pod \"iptables-alerter-fthc7\" (UID: \"93f95dad-8a63-4554-991b-196c269595e8\") " pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.214312 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-etc-kubernetes\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.214312 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214269 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-os-release\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.214312 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214287 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-host\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214326 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-systemd-units\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214354 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-run-systemd\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214378 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-cni-netd\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214404 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-var-lib-cni-bin\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214429 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-var-lib-kubelet\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214448 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93f95dad-8a63-4554-991b-196c269595e8-host-slash\") pod \"iptables-alerter-fthc7\" (UID: \"93f95dad-8a63-4554-991b-196c269595e8\") " pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7qs\" (UniqueName: \"kubernetes.io/projected/e845a4eb-a61f-4035-8dce-2a2600cd7952-kube-api-access-qr7qs\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214486 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-run-ovn-kubernetes\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214491 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214501 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-cnibin\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214515 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-var-lib-kubelet\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214536 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-conf-dir\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214550 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-os-release\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214567 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-system-cni-dir\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214589 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-cni-bin\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.214600 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214603 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-lib-modules\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214629 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkc66\" (UniqueName: \"kubernetes.io/projected/b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9-kube-api-access-bkc66\") pod \"node-resolver-pl8t9\" (UID: \"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9\") " pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214645 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-var-lib-openvswitch\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214659 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-run-ovn\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214674 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47325823-e4e0-4aa6-80c0-fe794b86efbe-ovnkube-config\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214688 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-system-cni-dir\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214725 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sxc4\" (UniqueName: \"kubernetes.io/projected/b83152b7-d9db-42fc-8792-f6d38f731245-kube-api-access-7sxc4\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214743 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/387a88ac-aa70-4b5f-8f71-8ff684c047a8-konnectivity-ca\") pod \"konnectivity-agent-mw8jp\" (UID: \"387a88ac-aa70-4b5f-8f71-8ff684c047a8\") " pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-device-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214783 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da43f1b5-a039-43c2-be38-894330b7745e-host\") pod \"node-ca-b9smv\" (UID: \"da43f1b5-a039-43c2-be38-894330b7745e\") " pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214805 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-kubelet-config\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214818 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-run-netns\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-hostroot\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214870 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-sysctl-d\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214884 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-sysctl-conf\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214906 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214936 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d027c33-cd10-4a2c-8a9f-90396b81d639-cni-binary-copy\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.215068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214950 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-modprobe-d\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214964 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9-tmp-dir\") pod \"node-resolver-pl8t9\" (UID: \"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9\") " pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214984 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-run-openvswitch\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.214988 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215009 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-systemd\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-sys\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215040 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215056 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47325823-e4e0-4aa6-80c0-fe794b86efbe-ovnkube-script-lib\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215072 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-run-k8s-cni-cncf-io\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215105 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-tuned\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215240 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-dbus\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215288 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47325823-e4e0-4aa6-80c0-fe794b86efbe-ovn-node-metrics-cert\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215308 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-run-multus-certs\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215311 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-device-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.215640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.215324 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgtx9\" (UniqueName: \"kubernetes.io/projected/5d027c33-cd10-4a2c-8a9f-90396b81d639-kube-api-access-sgtx9\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.216754 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.216715 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-system-cni-dir\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.217320 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.217279 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/387a88ac-aa70-4b5f-8f71-8ff684c047a8-konnectivity-ca\") pod \"konnectivity-agent-mw8jp\" (UID: \"387a88ac-aa70-4b5f-8f71-8ff684c047a8\") " pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:01.217411 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.217365 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.217516 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-kubernetes\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.217887 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.217996 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218040 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-socket-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218076 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-sys-fs\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218135 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-cnibin\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218171 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-kubelet\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218204 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47325823-e4e0-4aa6-80c0-fe794b86efbe-env-overrides\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218237 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g8dr\" (UniqueName: \"kubernetes.io/projected/47325823-e4e0-4aa6-80c0-fe794b86efbe-kube-api-access-5g8dr\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218421 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-socket-dir-parent\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218458 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-etc-selinux\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218490 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-etc-openvswitch\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218523 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218554 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-node-log\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218582 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/387a88ac-aa70-4b5f-8f71-8ff684c047a8-agent-certs\") pod \"konnectivity-agent-mw8jp\" (UID: \"387a88ac-aa70-4b5f-8f71-8ff684c047a8\") " pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218613 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-registration-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.220758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218641 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th47s\" (UniqueName: \"kubernetes.io/projected/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-kube-api-access-th47s\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218950 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.218993 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-var-lib-cni-multus\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219026 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-sysconfig\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219058 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qg5r\" (UniqueName: \"kubernetes.io/projected/da43f1b5-a039-43c2-be38-894330b7745e-kube-api-access-7qg5r\") pod \"node-ca-b9smv\" (UID: \"da43f1b5-a039-43c2-be38-894330b7745e\") " pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219087 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-run-netns\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219104 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-run\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219127 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7643ad2c-e05c-4180-b99c-1b419e46540e-tmp\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219161 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjt7p\" (UniqueName: \"kubernetes.io/projected/7643ad2c-e05c-4180-b99c-1b419e46540e-kube-api-access-zjt7p\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219207 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbg5q\" (UniqueName: \"kubernetes.io/projected/93f95dad-8a63-4554-991b-196c269595e8-kube-api-access-vbg5q\") pod \"iptables-alerter-fthc7\" (UID: \"93f95dad-8a63-4554-991b-196c269595e8\") " pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219242 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9-hosts-file\") pod \"node-resolver-pl8t9\" (UID: \"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9\") " pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219275 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219309 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-log-socket\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219340 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-cni-dir\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-daemon-config\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219404 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da43f1b5-a039-43c2-be38-894330b7745e-serviceca\") pod \"node-ca-b9smv\" (UID: \"da43f1b5-a039-43c2-be38-894330b7745e\") " pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.221511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.219730 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-sys-fs\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.222588 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.219829 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs podName:b83152b7-d9db-42fc-8792-f6d38f731245 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:01.719790664 +0000 UTC m=+2.952518324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs") pod "network-metrics-daemon-4hnrl" (UID: "b83152b7-d9db-42fc-8792-f6d38f731245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:01.222588 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.220065 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-socket-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.222588 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.220109 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-cnibin\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.222588 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.220206 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-etc-selinux\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.222588 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.220498 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 15:36:01.222588 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.220938 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-registration-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.223005 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.222979 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.223087 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.223064 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e845a4eb-a61f-4035-8dce-2a2600cd7952-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.223751 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.223238 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-cni-binary-copy\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.227559 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.227540 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sxc4\" (UniqueName: \"kubernetes.io/projected/b83152b7-d9db-42fc-8792-f6d38f731245-kube-api-access-7sxc4\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:01.227661 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.227617 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/387a88ac-aa70-4b5f-8f71-8ff684c047a8-agent-certs\") pod \"konnectivity-agent-mw8jp\" (UID: \"387a88ac-aa70-4b5f-8f71-8ff684c047a8\") " pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:01.228195 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.228152 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7qs\" (UniqueName: \"kubernetes.io/projected/e845a4eb-a61f-4035-8dce-2a2600cd7952-kube-api-access-qr7qs\") pod \"aws-ebs-csi-driver-node-7f2s5\" (UID: \"e845a4eb-a61f-4035-8dce-2a2600cd7952\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.229356 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.229313 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th47s\" (UniqueName: \"kubernetes.io/projected/d7f7dfa0-21b2-44d9-b821-7d7af91d493c-kube-api-access-th47s\") pod \"multus-additional-cni-plugins-s7vsx\" (UID: \"d7f7dfa0-21b2-44d9-b821-7d7af91d493c\") " pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.249270 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.249224 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" event={"ID":"d3350393ac72409ced117e07620daf2a","Type":"ContainerStarted","Data":"3bbc52fec2c7c24df60403f1ee9e4b7099a82ed9aa9ffb284d41606d13bdf30c"} Jul 24 15:36:01.250612 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.250563 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal" event={"ID":"fcc72598c4b952733bb79430abb4e958","Type":"ContainerStarted","Data":"3e838583f568786f6d403005675613e28330fd623c64974bc75741e500a802a1"} Jul 24 15:36:01.323807 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323713 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93f95dad-8a63-4554-991b-196c269595e8-iptables-alerter-script\") pod \"iptables-alerter-fthc7\" (UID: \"93f95dad-8a63-4554-991b-196c269595e8\") " pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.323807 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323761 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-etc-kubernetes\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.323807 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323781 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-host\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.323807 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323805 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-systemd-units\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323819 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-run-systemd\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-cni-netd\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323900 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-cni-netd\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323901 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-host\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323917 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-etc-kubernetes\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323901 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-systemd-units\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323928 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-var-lib-cni-bin\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-var-lib-kubelet\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324089 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-var-lib-kubelet\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324086 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93f95dad-8a63-4554-991b-196c269595e8-host-slash\") pod \"iptables-alerter-fthc7\" (UID: \"93f95dad-8a63-4554-991b-196c269595e8\") " pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323946 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-run-systemd\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.323955 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-var-lib-cni-bin\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324123 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-run-ovn-kubernetes\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324126 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324131 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93f95dad-8a63-4554-991b-196c269595e8-host-slash\") pod \"iptables-alerter-fthc7\" (UID: \"93f95dad-8a63-4554-991b-196c269595e8\") " pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324148 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-cnibin\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324172 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-var-lib-kubelet\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324205 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-conf-dir\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324227 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-cnibin\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324238 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-os-release\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324261 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-run-ovn-kubernetes\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324263 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-cni-bin\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324279 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/93f95dad-8a63-4554-991b-196c269595e8-iptables-alerter-script\") pod \"iptables-alerter-fthc7\" (UID: \"93f95dad-8a63-4554-991b-196c269595e8\") " pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324295 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-lib-modules\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324303 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-var-lib-kubelet\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324311 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkc66\" (UniqueName: \"kubernetes.io/projected/b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9-kube-api-access-bkc66\") pod \"node-resolver-pl8t9\" (UID: \"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9\") " pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324327 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-var-lib-openvswitch\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324336 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-cni-bin\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324341 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-run-ovn\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324356 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47325823-e4e0-4aa6-80c0-fe794b86efbe-ovnkube-config\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324370 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-system-cni-dir\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324376 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-conf-dir\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.324906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324386 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da43f1b5-a039-43c2-be38-894330b7745e-host\") pod \"node-ca-b9smv\" (UID: \"da43f1b5-a039-43c2-be38-894330b7745e\") " pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-kubelet-config\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324423 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-run-netns\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-hostroot\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324444 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-os-release\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324434 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-lib-modules\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324460 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-sysctl-d\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324482 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-sysctl-conf\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324503 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d027c33-cd10-4a2c-8a9f-90396b81d639-cni-binary-copy\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324520 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-modprobe-d\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324535 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9-tmp-dir\") pod \"node-resolver-pl8t9\" (UID: \"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9\") " pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324549 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-run-openvswitch\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324578 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-run-openvswitch\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324613 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/da43f1b5-a039-43c2-be38-894330b7745e-host\") pod \"node-ca-b9smv\" (UID: \"da43f1b5-a039-43c2-be38-894330b7745e\") " pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324645 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-kubelet-config\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324516 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-var-lib-openvswitch\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324495 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-system-cni-dir\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324725 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-run-ovn\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.325792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324792 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-run-netns\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324827 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-sysctl-d\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324846 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-hostroot\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324930 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-sysctl-conf\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.324936 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-modprobe-d\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325079 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-systemd\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325106 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-sys\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325131 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325139 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/47325823-e4e0-4aa6-80c0-fe794b86efbe-ovnkube-config\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325156 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9-tmp-dir\") pod \"node-resolver-pl8t9\" (UID: \"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9\") " pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325157 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47325823-e4e0-4aa6-80c0-fe794b86efbe-ovnkube-script-lib\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325206 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325220 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-run-k8s-cni-cncf-io\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325247 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-tuned\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325255 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-systemd\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-dbus\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325297 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-run-k8s-cni-cncf-io\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325297 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47325823-e4e0-4aa6-80c0-fe794b86efbe-ovn-node-metrics-cert\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.326663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325331 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-run-multus-certs\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325356 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgtx9\" (UniqueName: \"kubernetes.io/projected/5d027c33-cd10-4a2c-8a9f-90396b81d639-kube-api-access-sgtx9\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-kubernetes\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-kubelet\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325467 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47325823-e4e0-4aa6-80c0-fe794b86efbe-env-overrides\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8dr\" (UniqueName: \"kubernetes.io/projected/47325823-e4e0-4aa6-80c0-fe794b86efbe-kube-api-access-5g8dr\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325516 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-socket-dir-parent\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-etc-openvswitch\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325574 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-node-log\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325645 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325670 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-var-lib-cni-multus\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325720 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/47325823-e4e0-4aa6-80c0-fe794b86efbe-ovnkube-script-lib\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325743 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-sysconfig\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325762 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5d027c33-cd10-4a2c-8a9f-90396b81d639-cni-binary-copy\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325785 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-sysconfig\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325793 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-sys\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.327569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325811 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-run-multus-certs\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325825 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qg5r\" (UniqueName: \"kubernetes.io/projected/da43f1b5-a039-43c2-be38-894330b7745e-kube-api-access-7qg5r\") pod \"node-ca-b9smv\" (UID: \"da43f1b5-a039-43c2-be38-894330b7745e\") " pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325843 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-kubernetes\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325853 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-run-netns\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325868 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-kubelet\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325877 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-run\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7643ad2c-e05c-4180-b99c-1b419e46540e-tmp\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325943 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjt7p\" (UniqueName: \"kubernetes.io/projected/7643ad2c-e05c-4180-b99c-1b419e46540e-kube-api-access-zjt7p\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325968 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbg5q\" (UniqueName: \"kubernetes.io/projected/93f95dad-8a63-4554-991b-196c269595e8-kube-api-access-vbg5q\") pod \"iptables-alerter-fthc7\" (UID: \"93f95dad-8a63-4554-991b-196c269595e8\") " pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.325991 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9-hosts-file\") pod \"node-resolver-pl8t9\" (UID: \"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9\") " pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326017 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-log-socket\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326042 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-cni-dir\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326065 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-daemon-config\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da43f1b5-a039-43c2-be38-894330b7745e-serviceca\") pod \"node-ca-b9smv\" (UID: \"da43f1b5-a039-43c2-be38-894330b7745e\") " pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326104 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/47325823-e4e0-4aa6-80c0-fe794b86efbe-env-overrides\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326119 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-slash\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326191 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-host-slash\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326357 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-dbus\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.328399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326364 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-socket-dir-parent\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326486 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-cni-dir\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326593 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-var-lib-cni-multus\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326715 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-node-log\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326762 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-etc-openvswitch\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.326827 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.326968 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5d027c33-cd10-4a2c-8a9f-90396b81d639-multus-daemon-config\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.327038 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d027c33-cd10-4a2c-8a9f-90396b81d639-host-run-netns\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.327127 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7643ad2c-e05c-4180-b99c-1b419e46540e-run\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.327197 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret podName:6a466ffd-d49f-45ea-ba0e-27bcdaf53f85 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:01.82717784 +0000 UTC m=+3.059905511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret") pod "global-pull-secret-syncer-rbhv6" (UID: "6a466ffd-d49f-45ea-ba0e-27bcdaf53f85") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.327404 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9-hosts-file\") pod \"node-resolver-pl8t9\" (UID: \"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9\") " pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.327587 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/47325823-e4e0-4aa6-80c0-fe794b86efbe-log-socket\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.327890 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/47325823-e4e0-4aa6-80c0-fe794b86efbe-ovn-node-metrics-cert\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.329218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.328563 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7643ad2c-e05c-4180-b99c-1b419e46540e-etc-tuned\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.329845 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.329257 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7643ad2c-e05c-4180-b99c-1b419e46540e-tmp\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.332844 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.332824 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/da43f1b5-a039-43c2-be38-894330b7745e-serviceca\") pod \"node-ca-b9smv\" (UID: \"da43f1b5-a039-43c2-be38-894330b7745e\") " pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.334234 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.333979 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:36:01.334234 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.334000 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:36:01.334234 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.334014 2571 projected.go:196] Error preparing data for projected volume kube-api-access-t2fdg for pod openshift-network-diagnostics/network-check-target-df4sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:01.334234 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.334066 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg podName:cf25c7d5-8abf-40e0-bf77-9611072a1d43 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:01.834050966 +0000 UTC m=+3.066778639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t2fdg" (UniqueName: "kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg") pod "network-check-target-df4sc" (UID: "cf25c7d5-8abf-40e0-bf77-9611072a1d43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:01.336640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.336585 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qg5r\" (UniqueName: \"kubernetes.io/projected/da43f1b5-a039-43c2-be38-894330b7745e-kube-api-access-7qg5r\") pod \"node-ca-b9smv\" (UID: \"da43f1b5-a039-43c2-be38-894330b7745e\") " pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.337446 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.337198 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8dr\" (UniqueName: \"kubernetes.io/projected/47325823-e4e0-4aa6-80c0-fe794b86efbe-kube-api-access-5g8dr\") pod \"ovnkube-node-czgtt\" (UID: \"47325823-e4e0-4aa6-80c0-fe794b86efbe\") " pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.337618 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.337599 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbg5q\" (UniqueName: \"kubernetes.io/projected/93f95dad-8a63-4554-991b-196c269595e8-kube-api-access-vbg5q\") pod \"iptables-alerter-fthc7\" (UID: \"93f95dad-8a63-4554-991b-196c269595e8\") " pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.338000 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.337953 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkc66\" (UniqueName: \"kubernetes.io/projected/b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9-kube-api-access-bkc66\") pod \"node-resolver-pl8t9\" (UID: \"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9\") " pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.338128 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.338109 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjt7p\" (UniqueName: \"kubernetes.io/projected/7643ad2c-e05c-4180-b99c-1b419e46540e-kube-api-access-zjt7p\") pod \"tuned-qqr6d\" (UID: \"7643ad2c-e05c-4180-b99c-1b419e46540e\") " pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.340296 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.340279 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgtx9\" (UniqueName: \"kubernetes.io/projected/5d027c33-cd10-4a2c-8a9f-90396b81d639-kube-api-access-sgtx9\") pod \"multus-dzpj8\" (UID: \"5d027c33-cd10-4a2c-8a9f-90396b81d639\") " pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.412348 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.412309 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" Jul 24 15:36:01.420201 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.420174 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:01.431901 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.431873 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" Jul 24 15:36:01.438543 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.438521 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" Jul 24 15:36:01.446316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.446286 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b9smv" Jul 24 15:36:01.455931 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.455911 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fthc7" Jul 24 15:36:01.463666 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.463642 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:01.471462 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.471442 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzpj8" Jul 24 15:36:01.477024 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.477005 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pl8t9" Jul 24 15:36:01.729153 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.729077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:01.729305 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.729254 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:01.729365 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.729334 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs podName:b83152b7-d9db-42fc-8792-f6d38f731245 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:02.729313594 +0000 UTC m=+3.962041254 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs") pod "network-metrics-daemon-4hnrl" (UID: "b83152b7-d9db-42fc-8792-f6d38f731245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:01.829485 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.829439 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:01.829676 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.829596 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:01.829676 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.829672 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret podName:6a466ffd-d49f-45ea-ba0e-27bcdaf53f85 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:02.829656792 +0000 UTC m=+4.062384452 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret") pod "global-pull-secret-syncer-rbhv6" (UID: "6a466ffd-d49f-45ea-ba0e-27bcdaf53f85") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:01.930628 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:01.930587 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:01.930839 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.930756 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:36:01.930839 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.930775 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:36:01.930839 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.930786 2571 projected.go:196] Error preparing data for projected volume kube-api-access-t2fdg for pod openshift-network-diagnostics/network-check-target-df4sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:01.930975 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:01.930862 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg podName:cf25c7d5-8abf-40e0-bf77-9611072a1d43 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:02.930843619 +0000 UTC m=+4.163571277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t2fdg" (UniqueName: "kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg") pod "network-check-target-df4sc" (UID: "cf25c7d5-8abf-40e0-bf77-9611072a1d43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:02.042161 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:02.042133 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f7dfa0_21b2_44d9_b821_7d7af91d493c.slice/crio-1338a00803afd56ba1fd791084db660d7bfa38a619f3143eccd3069c4c332ca4 WatchSource:0}: Error finding container 1338a00803afd56ba1fd791084db660d7bfa38a619f3143eccd3069c4c332ca4: Status 404 returned error can't find the container with id 1338a00803afd56ba1fd791084db660d7bfa38a619f3143eccd3069c4c332ca4 Jul 24 15:36:02.062113 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:02.061952 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda43f1b5_a039_43c2_be38_894330b7745e.slice/crio-6f7b464ace7e2c16230768bd6bfa2cce5cd52ab9ec7ebd3c595f3ed3a6d36a92 WatchSource:0}: Error finding container 6f7b464ace7e2c16230768bd6bfa2cce5cd52ab9ec7ebd3c595f3ed3a6d36a92: Status 404 returned error can't find the container with id 6f7b464ace7e2c16230768bd6bfa2cce5cd52ab9ec7ebd3c595f3ed3a6d36a92 Jul 24 15:36:02.067609 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:02.067585 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f95dad_8a63_4554_991b_196c269595e8.slice/crio-0b636cdcf8875ef0f675c072a61ac2644ee5ee39bad83680a12ac43052c39769 WatchSource:0}: Error finding container 0b636cdcf8875ef0f675c072a61ac2644ee5ee39bad83680a12ac43052c39769: Status 404 returned error can't find the container with id 0b636cdcf8875ef0f675c072a61ac2644ee5ee39bad83680a12ac43052c39769 Jul 24 15:36:02.068324 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:02.068222 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387a88ac_aa70_4b5f_8f71_8ff684c047a8.slice/crio-2d48bf744085039f707d9c648815f694e3587d22b1f5b35b5753bc8ae3489865 WatchSource:0}: Error finding container 2d48bf744085039f707d9c648815f694e3587d22b1f5b35b5753bc8ae3489865: Status 404 returned error can't find the container with id 2d48bf744085039f707d9c648815f694e3587d22b1f5b35b5753bc8ae3489865 Jul 24 15:36:02.069810 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:02.069678 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d027c33_cd10_4a2c_8a9f_90396b81d639.slice/crio-d4eb186897f99027ad3d6672cf09f2d4b8105e5238c333876be9276b4c3ad884 WatchSource:0}: Error finding container d4eb186897f99027ad3d6672cf09f2d4b8105e5238c333876be9276b4c3ad884: Status 404 returned error can't find the container with id d4eb186897f99027ad3d6672cf09f2d4b8105e5238c333876be9276b4c3ad884 Jul 24 15:36:02.072904 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:02.070802 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47325823_e4e0_4aa6_80c0_fe794b86efbe.slice/crio-a6f0455e03c598647a67ab63792eb87657fb82b3b97f97073a05c21d0b9df740 WatchSource:0}: Error finding container a6f0455e03c598647a67ab63792eb87657fb82b3b97f97073a05c21d0b9df740: Status 404 returned error can't find the container with id a6f0455e03c598647a67ab63792eb87657fb82b3b97f97073a05c21d0b9df740 Jul 24 15:36:02.153615 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.153439 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:31:01 +0000 UTC" deadline="2028-01-07 14:51:09.896469607 +0000 UTC" Jul 24 15:36:02.153615 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.153611 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12767h15m7.742862275s" Jul 24 15:36:02.245305 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.245274 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:02.245468 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:02.245395 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:02.254687 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.254641 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" event={"ID":"7643ad2c-e05c-4180-b99c-1b419e46540e","Type":"ContainerStarted","Data":"23f0edf83fef35a0a1c1f2d29109945cc7133e591649cc57843c0ec5f6ed0c34"} Jul 24 15:36:02.255955 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.255923 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" event={"ID":"e845a4eb-a61f-4035-8dce-2a2600cd7952","Type":"ContainerStarted","Data":"d0d8a084d5e9ddaaf8f738ee8ddfbb26ffbe89d3f82b5d25d26478f75b5814d9"} Jul 24 15:36:02.257219 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.257187 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerStarted","Data":"a6f0455e03c598647a67ab63792eb87657fb82b3b97f97073a05c21d0b9df740"} Jul 24 15:36:02.258768 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.258725 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mw8jp" event={"ID":"387a88ac-aa70-4b5f-8f71-8ff684c047a8","Type":"ContainerStarted","Data":"2d48bf744085039f707d9c648815f694e3587d22b1f5b35b5753bc8ae3489865"} Jul 24 15:36:02.260147 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.260121 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fthc7" event={"ID":"93f95dad-8a63-4554-991b-196c269595e8","Type":"ContainerStarted","Data":"0b636cdcf8875ef0f675c072a61ac2644ee5ee39bad83680a12ac43052c39769"} Jul 24 15:36:02.261330 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.261295 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b9smv" event={"ID":"da43f1b5-a039-43c2-be38-894330b7745e","Type":"ContainerStarted","Data":"6f7b464ace7e2c16230768bd6bfa2cce5cd52ab9ec7ebd3c595f3ed3a6d36a92"} Jul 24 15:36:02.262427 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.262402 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" event={"ID":"d7f7dfa0-21b2-44d9-b821-7d7af91d493c","Type":"ContainerStarted","Data":"1338a00803afd56ba1fd791084db660d7bfa38a619f3143eccd3069c4c332ca4"} Jul 24 15:36:02.264522 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.264495 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal" event={"ID":"fcc72598c4b952733bb79430abb4e958","Type":"ContainerStarted","Data":"62595af0a8e9520f4cbf29a7d7fbe0e11b336f1c9c225659ad73aa8c7ea830e4"} Jul 24 15:36:02.265731 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.265622 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pl8t9" event={"ID":"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9","Type":"ContainerStarted","Data":"7763f0061699969fd8b35153019c70e7a19e8041903fb85c111f29666081716e"} Jul 24 15:36:02.266767 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.266743 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzpj8" event={"ID":"5d027c33-cd10-4a2c-8a9f-90396b81d639","Type":"ContainerStarted","Data":"d4eb186897f99027ad3d6672cf09f2d4b8105e5238c333876be9276b4c3ad884"} Jul 24 15:36:02.280826 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.280778 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-156.ec2.internal" podStartSLOduration=2.2807622260000002 podStartE2EDuration="2.280762226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:02.280600761 +0000 UTC m=+3.513328439" watchObservedRunningTime="2026-07-24 15:36:02.280762226 +0000 UTC m=+3.513489908" Jul 24 15:36:02.738446 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.738353 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:02.738588 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:02.738538 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:02.738651 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:02.738604 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs podName:b83152b7-d9db-42fc-8792-f6d38f731245 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:04.738586748 +0000 UTC m=+5.971314418 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs") pod "network-metrics-daemon-4hnrl" (UID: "b83152b7-d9db-42fc-8792-f6d38f731245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:02.838814 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.838773 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:02.839038 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:02.838996 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:02.839121 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:02.839064 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret podName:6a466ffd-d49f-45ea-ba0e-27bcdaf53f85 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:04.839045008 +0000 UTC m=+6.071772667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret") pod "global-pull-secret-syncer-rbhv6" (UID: "6a466ffd-d49f-45ea-ba0e-27bcdaf53f85") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:02.939484 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:02.939446 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:02.939738 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:02.939629 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:36:02.939738 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:02.939651 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:36:02.939738 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:02.939666 2571 projected.go:196] Error preparing data for projected volume kube-api-access-t2fdg for pod openshift-network-diagnostics/network-check-target-df4sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:02.939936 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:02.939743 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg podName:cf25c7d5-8abf-40e0-bf77-9611072a1d43 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:04.939723413 +0000 UTC m=+6.172451070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t2fdg" (UniqueName: "kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg") pod "network-check-target-df4sc" (UID: "cf25c7d5-8abf-40e0-bf77-9611072a1d43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:03.155634 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:03.155529 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:31:01 +0000 UTC" deadline="2028-03-16 08:04:09.542524053 +0000 UTC" Jul 24 15:36:03.155634 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:03.155573 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14416h28m6.386955776s" Jul 24 15:36:03.246330 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:03.245817 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:03.246330 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:03.245957 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:03.246904 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:03.246715 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:03.246904 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:03.246830 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:04.245542 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:04.245047 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:04.245542 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:04.245170 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:04.757128 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:04.757095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:04.757416 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:04.757397 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:04.757476 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:04.757466 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs podName:b83152b7-d9db-42fc-8792-f6d38f731245 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:08.757447532 +0000 UTC m=+9.990175190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs") pod "network-metrics-daemon-4hnrl" (UID: "b83152b7-d9db-42fc-8792-f6d38f731245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:04.858200 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:04.858169 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:04.858392 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:04.858374 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:04.858471 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:04.858443 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret podName:6a466ffd-d49f-45ea-ba0e-27bcdaf53f85 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:08.85842312 +0000 UTC m=+10.091150789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret") pod "global-pull-secret-syncer-rbhv6" (UID: "6a466ffd-d49f-45ea-ba0e-27bcdaf53f85") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:04.958992 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:04.958957 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:04.959171 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:04.959133 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:36:04.959171 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:04.959151 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:36:04.959171 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:04.959162 2571 projected.go:196] Error preparing data for projected volume kube-api-access-t2fdg for pod openshift-network-diagnostics/network-check-target-df4sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:04.959314 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:04.959215 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg podName:cf25c7d5-8abf-40e0-bf77-9611072a1d43 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:08.959197367 +0000 UTC m=+10.191925048 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t2fdg" (UniqueName: "kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg") pod "network-check-target-df4sc" (UID: "cf25c7d5-8abf-40e0-bf77-9611072a1d43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:05.247279 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:05.246591 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:05.247279 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:05.246744 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:05.247279 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:05.247192 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:05.247919 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:05.247300 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:05.287862 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:05.287818 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" event={"ID":"d3350393ac72409ced117e07620daf2a","Type":"ContainerStarted","Data":"4ea1796b540f54d7d9b97652efa5f8256a40d016d28e59238cd7193650cd6801"} Jul 24 15:36:06.245901 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:06.245397 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:06.245901 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:06.245525 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:06.293150 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:06.293112 2571 generic.go:356] "Generic (PLEG): container finished" podID="d3350393ac72409ced117e07620daf2a" containerID="4ea1796b540f54d7d9b97652efa5f8256a40d016d28e59238cd7193650cd6801" exitCode=0 Jul 24 15:36:06.293488 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:06.293174 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" event={"ID":"d3350393ac72409ced117e07620daf2a","Type":"ContainerDied","Data":"4ea1796b540f54d7d9b97652efa5f8256a40d016d28e59238cd7193650cd6801"} Jul 24 15:36:07.248604 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:07.248572 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:07.248800 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:07.248717 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:07.249104 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:07.249086 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:07.249215 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:07.249189 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:08.244949 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:08.244906 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:08.245379 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:08.245022 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:08.787932 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:08.787890 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:08.788137 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:08.788079 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:08.788201 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:08.788148 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs podName:b83152b7-d9db-42fc-8792-f6d38f731245 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:16.788129349 +0000 UTC m=+18.020857012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs") pod "network-metrics-daemon-4hnrl" (UID: "b83152b7-d9db-42fc-8792-f6d38f731245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:08.889140 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:08.889105 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:08.889306 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:08.889257 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:08.889373 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:08.889320 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret podName:6a466ffd-d49f-45ea-ba0e-27bcdaf53f85 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:16.889304385 +0000 UTC m=+18.122032047 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret") pod "global-pull-secret-syncer-rbhv6" (UID: "6a466ffd-d49f-45ea-ba0e-27bcdaf53f85") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:08.989566 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:08.989521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:08.989826 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:08.989779 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:36:08.989826 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:08.989807 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:36:08.989826 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:08.989820 2571 projected.go:196] Error preparing data for projected volume kube-api-access-t2fdg for pod openshift-network-diagnostics/network-check-target-df4sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:08.989998 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:08.989899 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg podName:cf25c7d5-8abf-40e0-bf77-9611072a1d43 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:16.989881072 +0000 UTC m=+18.222608730 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t2fdg" (UniqueName: "kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg") pod "network-check-target-df4sc" (UID: "cf25c7d5-8abf-40e0-bf77-9611072a1d43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:09.245390 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:09.245356 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:09.245882 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:09.245468 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:09.245943 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:09.245889 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:09.246013 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:09.245988 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:10.245308 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:10.245214 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:10.245476 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:10.245353 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:11.245473 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:11.245438 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:11.245959 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:11.245438 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:11.245959 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:11.245586 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:11.245959 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:11.245621 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:12.245104 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:12.245068 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:12.245329 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:12.245196 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:13.245448 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.245241 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:13.245920 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.245241 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:13.245920 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:13.245557 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:13.245920 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:13.245633 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:13.310723 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.309828 2571 generic.go:356] "Generic (PLEG): container finished" podID="d7f7dfa0-21b2-44d9-b821-7d7af91d493c" containerID="a71647ebfdbfee741bab83eb55d961c37ffbc7a99bfac8cd0ef188a3189cd66c" exitCode=0 Jul 24 15:36:13.310723 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.309876 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" event={"ID":"d7f7dfa0-21b2-44d9-b821-7d7af91d493c","Type":"ContainerDied","Data":"a71647ebfdbfee741bab83eb55d961c37ffbc7a99bfac8cd0ef188a3189cd66c"} Jul 24 15:36:13.311970 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.311944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pl8t9" event={"ID":"b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9","Type":"ContainerStarted","Data":"6a4f92fec3c371fea519d56aa9a678f01730ee21f5cc3d228a7f9879adf22fee"} Jul 24 15:36:13.314274 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.314250 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" event={"ID":"d3350393ac72409ced117e07620daf2a","Type":"ContainerStarted","Data":"811fdeda13a0ae34c0022923b786d87ea46255ddfe0d567862762e1428bf351c"} Jul 24 15:36:13.315937 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.315910 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" event={"ID":"7643ad2c-e05c-4180-b99c-1b419e46540e","Type":"ContainerStarted","Data":"bc4929d0e4138589163d33bc092e57bec38cd0f1a54d172f6f357fb99f51ae08"} Jul 24 15:36:13.318501 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.318469 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" event={"ID":"e845a4eb-a61f-4035-8dce-2a2600cd7952","Type":"ContainerStarted","Data":"1c4465b5f1e89e31aef1ff46154fe3438742fb5c7cf17eda18911a81f5c36749"} Jul 24 15:36:13.320053 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.320019 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mw8jp" event={"ID":"387a88ac-aa70-4b5f-8f71-8ff684c047a8","Type":"ContainerStarted","Data":"efb6a115b00897a17ce8f263e216de617dedf9e3d77dee2858151367d49b5797"} Jul 24 15:36:13.321690 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.321669 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b9smv" event={"ID":"da43f1b5-a039-43c2-be38-894330b7745e","Type":"ContainerStarted","Data":"b98444e8429509069744df291473ff95fd551e47bb1380bb7a7bc6bbbd84edeb"} Jul 24 15:36:13.347491 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.347429 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mw8jp" podStartSLOduration=4.139722891 podStartE2EDuration="14.347408247s" totalImagesPullingTime="10.207685356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:13.346866008 +0000 UTC m=+14.579593685" watchObservedRunningTime="2026-07-24 15:36:13.347408247 +0000 UTC m=+14.580135926" Jul 24 15:36:13.363632 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.363572 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qqr6d" podStartSLOduration=4.109330558 podStartE2EDuration="14.363552467s" totalImagesPullingTime="10.254221909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:13.363205302 +0000 UTC m=+14.595932981" watchObservedRunningTime="2026-07-24 15:36:13.363552467 +0000 UTC m=+14.596280147" Jul 24 15:36:13.377480 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.377417 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b9smv" podStartSLOduration=4.14583795 podStartE2EDuration="14.377389558s" totalImagesPullingTime="10.231551608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:13.376415379 +0000 UTC m=+14.609143058" watchObservedRunningTime="2026-07-24 15:36:13.377389558 +0000 UTC m=+14.610117237" Jul 24 15:36:13.394607 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.394504 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pl8t9" podStartSLOduration=4.176801059 podStartE2EDuration="14.394481524s" totalImagesPullingTime="10.217680465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:13.393954213 +0000 UTC m=+14.626681891" watchObservedRunningTime="2026-07-24 15:36:13.394481524 +0000 UTC m=+14.627209204" Jul 24 15:36:13.418504 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:13.418454 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-156.ec2.internal" podStartSLOduration=13.418434273999999 podStartE2EDuration="13.418434274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:13.417599266 +0000 UTC m=+14.650326943" watchObservedRunningTime="2026-07-24 15:36:13.418434274 +0000 UTC m=+14.651161952" Jul 24 15:36:14.244872 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:14.244843 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:14.245027 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:14.244980 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:14.326335 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:14.326296 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fthc7" event={"ID":"93f95dad-8a63-4554-991b-196c269595e8","Type":"ContainerStarted","Data":"d1863028f2a98ad88e54b4aac8b468f71c0be7cec5e5e4fbe54a400b9ec5bb87"} Jul 24 15:36:15.244645 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:15.244618 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:15.244840 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:15.244746 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:15.245113 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:15.245087 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:15.245206 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:15.245191 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:16.101060 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:16.101024 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:16.101627 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:16.101603 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:16.118212 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:16.118165 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fthc7" podStartSLOduration=6.885049129 podStartE2EDuration="17.118149334s" totalImagesPullingTime="10.233100205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:14.342005226 +0000 UTC m=+15.574732913" watchObservedRunningTime="2026-07-24 15:36:16.118149334 +0000 UTC m=+17.350877009" Jul 24 15:36:16.245501 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:16.245265 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:16.245798 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:16.245468 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:16.852248 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:16.852206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:16.852440 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:16.852346 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:16.852440 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:16.852436 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs podName:b83152b7-d9db-42fc-8792-f6d38f731245 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:32.852406809 +0000 UTC m=+34.085134469 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs") pod "network-metrics-daemon-4hnrl" (UID: "b83152b7-d9db-42fc-8792-f6d38f731245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:16.953024 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:16.952980 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:16.953172 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:16.953147 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:16.953246 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:16.953233 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret podName:6a466ffd-d49f-45ea-ba0e-27bcdaf53f85 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:32.953212612 +0000 UTC m=+34.185940279 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret") pod "global-pull-secret-syncer-rbhv6" (UID: "6a466ffd-d49f-45ea-ba0e-27bcdaf53f85") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:17.053534 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:17.053492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:17.053741 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:17.053677 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:36:17.053741 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:17.053712 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:36:17.053741 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:17.053725 2571 projected.go:196] Error preparing data for projected volume kube-api-access-t2fdg for pod openshift-network-diagnostics/network-check-target-df4sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:17.053914 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:17.053784 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg podName:cf25c7d5-8abf-40e0-bf77-9611072a1d43 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:33.053766033 +0000 UTC m=+34.286493689 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t2fdg" (UniqueName: "kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg") pod "network-check-target-df4sc" (UID: "cf25c7d5-8abf-40e0-bf77-9611072a1d43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:17.245414 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:17.245383 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:17.246093 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:17.245521 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:17.246093 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:17.245558 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:17.246093 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:17.245657 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:18.245339 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:18.245302 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:18.245468 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:18.245412 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:19.245680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:19.245643 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:19.246134 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:19.245809 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:19.246134 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:19.245906 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:19.246134 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:19.246037 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:20.244401 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:20.244367 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:20.244555 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:20.244482 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:20.597943 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:20.597847 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:20.598480 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:20.598014 2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 24 15:36:20.598480 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:20.598430 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mw8jp" Jul 24 15:36:21.244474 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:21.244409 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:21.244474 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:21.244459 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:21.244766 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:21.244575 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:21.244766 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:21.244740 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:22.244874 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:22.244843 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:22.245375 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:22.244967 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:23.245173 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:23.245140 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:23.245622 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:23.245296 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:23.245622 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:23.245350 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:23.245622 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:23.245465 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:24.245057 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:24.245027 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:24.245216 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:24.245136 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:24.646582 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:24.646421 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 15:36:25.179886 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.179567 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T15:36:24.646579691Z","UUID":"c0f0542d-0cb8-4aa4-ba8d-e32235ff574e","Handler":null,"Name":"","Endpoint":""} Jul 24 15:36:25.181288 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.181269 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 15:36:25.181399 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.181305 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 15:36:25.244577 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.244543 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:25.244577 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.244590 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:25.244840 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:25.244738 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:25.244900 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:25.244838 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:25.349996 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.349918 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" event={"ID":"e845a4eb-a61f-4035-8dce-2a2600cd7952","Type":"ContainerStarted","Data":"de21b9324f73a63759e006311ba7f1b9a39cac4584b02932a04737118bb89a29"} Jul 24 15:36:25.353258 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.353230 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:36:25.353626 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.353597 2571 generic.go:356] "Generic (PLEG): container finished" podID="47325823-e4e0-4aa6-80c0-fe794b86efbe" containerID="1894932997bbfd633ede2b1fdd43089ca34ab36a478c21d1d32e69800d170098" exitCode=1 Jul 24 15:36:25.353762 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.353680 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerStarted","Data":"a4421d7129cddbb799fe543167019963fd15cbb10700280522d1ed0ab74a4ceb"} Jul 24 15:36:25.353828 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.353787 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerStarted","Data":"bdad057b589e61b25718783815789f3fc69acda701f99d9e5f8ed43f222a22dd"} Jul 24 15:36:25.353887 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.353835 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerStarted","Data":"74dff27f605d85541a48f5b980a15a1ca986dec38202eb6fc813a7ffce05cc5e"} Jul 24 15:36:25.353887 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.353871 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerStarted","Data":"23ab4abfccb9cc27cbce04461e9e024021a2fb39e87482328c98e5dc520c2ff3"} Jul 24 15:36:25.353980 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.353918 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerDied","Data":"1894932997bbfd633ede2b1fdd43089ca34ab36a478c21d1d32e69800d170098"} Jul 24 15:36:25.353980 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.353969 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerStarted","Data":"7be882d9c26b829648a0e7e31f0840d8df7c593ce8730e04b0aac6b30d81dea7"} Jul 24 15:36:25.355724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.355686 2571 generic.go:356] "Generic (PLEG): container finished" podID="d7f7dfa0-21b2-44d9-b821-7d7af91d493c" containerID="a83ed7172a58a4f326bedf79e0059cf9e98d310b577cd376a9f1779513237a19" exitCode=0 Jul 24 15:36:25.355830 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.355734 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" event={"ID":"d7f7dfa0-21b2-44d9-b821-7d7af91d493c","Type":"ContainerDied","Data":"a83ed7172a58a4f326bedf79e0059cf9e98d310b577cd376a9f1779513237a19"} Jul 24 15:36:25.357620 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.357600 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzpj8" event={"ID":"5d027c33-cd10-4a2c-8a9f-90396b81d639","Type":"ContainerStarted","Data":"ecae4d34bbc1984e014644e89f5896a78f620a8d413c319c78fb0f7b3a2eb61c"} Jul 24 15:36:25.409340 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:25.409286 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dzpj8" podStartSLOduration=3.998945711 podStartE2EDuration="26.409271596s" totalImagesPullingTime="22.410325885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:25.409168763 +0000 UTC m=+26.641896453" watchObservedRunningTime="2026-07-24 15:36:25.409271596 +0000 UTC m=+26.641999274" Jul 24 15:36:26.245325 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:26.245289 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:26.245435 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:26.245414 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:26.361992 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:26.361952 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" event={"ID":"e845a4eb-a61f-4035-8dce-2a2600cd7952","Type":"ContainerStarted","Data":"8d048d52102fe698b167e630bc77f108e839ccc065dd81965d858b7cf2df2b8d"} Jul 24 15:36:26.364031 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:26.364008 2571 generic.go:356] "Generic (PLEG): container finished" podID="d7f7dfa0-21b2-44d9-b821-7d7af91d493c" containerID="901989a1e3a8e54ed63b32f229d1a5434a27d8259b76e65e882da34f415820b4" exitCode=0 Jul 24 15:36:26.364171 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:26.364099 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" event={"ID":"d7f7dfa0-21b2-44d9-b821-7d7af91d493c","Type":"ContainerDied","Data":"901989a1e3a8e54ed63b32f229d1a5434a27d8259b76e65e882da34f415820b4"} Jul 24 15:36:26.379582 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:26.379543 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7f2s5" podStartSLOduration=4.420833701 podStartE2EDuration="27.379527517s" totalImagesPullingTime="22.958693816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:26.378850914 +0000 UTC m=+27.611578592" watchObservedRunningTime="2026-07-24 15:36:26.379527517 +0000 UTC m=+27.612255195" Jul 24 15:36:27.244609 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:27.244580 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:27.244778 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:27.244625 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:27.244778 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:27.244735 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:27.244876 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:27.244783 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:27.370382 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:27.370357 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:36:27.370884 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:27.370742 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerStarted","Data":"51a12bc9a6c56f133db647881105a6662963860c923c3f5368decf44163b99bb"} Jul 24 15:36:27.373178 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:27.373153 2571 generic.go:356] "Generic (PLEG): container finished" podID="d7f7dfa0-21b2-44d9-b821-7d7af91d493c" containerID="842b1f69fd59428096b18d55ec0a1800e09b64ea3888f9662a622bc7b835f82c" exitCode=0 Jul 24 15:36:27.373310 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:27.373241 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" event={"ID":"d7f7dfa0-21b2-44d9-b821-7d7af91d493c","Type":"ContainerDied","Data":"842b1f69fd59428096b18d55ec0a1800e09b64ea3888f9662a622bc7b835f82c"} Jul 24 15:36:28.244888 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:28.244851 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:28.245051 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:28.244966 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:29.245254 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:29.245222 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:29.245737 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:29.245331 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:29.245737 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:29.245386 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:29.245737 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:29.245490 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:30.245597 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:30.245352 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:30.246305 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:30.245686 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:30.382583 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:30.382559 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:36:30.382923 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:30.382900 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerStarted","Data":"16b56d36e0c82ed37296666367c87de933667bb17443c4357126c7865fea37b5"} Jul 24 15:36:30.383161 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:30.383143 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:30.383325 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:30.383310 2571 scope.go:117] "RemoveContainer" containerID="1894932997bbfd633ede2b1fdd43089ca34ab36a478c21d1d32e69800d170098" Jul 24 15:36:30.401163 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:30.401139 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:31.245463 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.245432 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:31.245661 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.245441 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:31.245661 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:31.245560 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:31.246064 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:31.245661 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:31.390438 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.390407 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:36:31.390875 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.390840 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" event={"ID":"47325823-e4e0-4aa6-80c0-fe794b86efbe","Type":"ContainerStarted","Data":"44d1fa68d99fc12c8f6bf6d9c5b04306165a647184f88f3b4bf5496c3f9c5c65"} Jul 24 15:36:31.391183 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.391163 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:31.391285 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.391248 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:31.411459 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.411432 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:36:31.466329 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.466277 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" podStartSLOduration=10.059300505 podStartE2EDuration="32.466257397s" totalImagesPullingTime="22.406956892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:31.42834199 +0000 UTC m=+32.661069669" watchObservedRunningTime="2026-07-24 15:36:31.466257397 +0000 UTC m=+32.698985087" Jul 24 15:36:31.688031 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.687932 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-df4sc"] Jul 24 15:36:31.688195 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.688101 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:31.688269 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:31.688196 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:31.690322 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.690295 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rbhv6"] Jul 24 15:36:31.690447 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.690401 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:31.690504 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:31.690478 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:31.693835 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.693812 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4hnrl"] Jul 24 15:36:31.693950 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:31.693919 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:31.694039 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:31.694018 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:32.861755 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:32.861730 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:32.862122 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:32.861888 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:32.862122 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:32.861956 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs podName:b83152b7-d9db-42fc-8792-f6d38f731245 nodeName:}" failed. No retries permitted until 2026-07-24 15:37:04.86193832 +0000 UTC m=+66.094665983 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs") pod "network-metrics-daemon-4hnrl" (UID: "b83152b7-d9db-42fc-8792-f6d38f731245") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:36:32.963104 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:32.963023 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:32.963239 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:32.963141 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:32.963239 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:32.963193 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret podName:6a466ffd-d49f-45ea-ba0e-27bcdaf53f85 nodeName:}" failed. No retries permitted until 2026-07-24 15:37:04.96318154 +0000 UTC m=+66.195909197 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret") pod "global-pull-secret-syncer-rbhv6" (UID: "6a466ffd-d49f-45ea-ba0e-27bcdaf53f85") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:36:33.064056 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:33.064019 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:33.064232 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:33.064155 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:36:33.064232 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:33.064170 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:36:33.064232 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:33.064180 2571 projected.go:196] Error preparing data for projected volume kube-api-access-t2fdg for pod openshift-network-diagnostics/network-check-target-df4sc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:33.064232 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:33.064231 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg podName:cf25c7d5-8abf-40e0-bf77-9611072a1d43 nodeName:}" failed. No retries permitted until 2026-07-24 15:37:05.064218041 +0000 UTC m=+66.296945698 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t2fdg" (UniqueName: "kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg") pod "network-check-target-df4sc" (UID: "cf25c7d5-8abf-40e0-bf77-9611072a1d43") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:36:33.245403 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:33.245364 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:33.245403 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:33.245398 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:33.245650 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:33.245368 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:33.245650 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:33.245487 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:33.245650 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:33.245547 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:33.245650 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:33.245632 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:33.399307 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:33.399271 2571 generic.go:356] "Generic (PLEG): container finished" podID="d7f7dfa0-21b2-44d9-b821-7d7af91d493c" containerID="b15fdc283224382194be258f46fd3b7ef856407e30a495bab0e742eed368d63e" exitCode=0 Jul 24 15:36:33.399498 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:33.399353 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" event={"ID":"d7f7dfa0-21b2-44d9-b821-7d7af91d493c","Type":"ContainerDied","Data":"b15fdc283224382194be258f46fd3b7ef856407e30a495bab0e742eed368d63e"} Jul 24 15:36:34.405217 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:34.404991 2571 generic.go:356] "Generic (PLEG): container finished" podID="d7f7dfa0-21b2-44d9-b821-7d7af91d493c" containerID="02ca6347db57f9cda3c832d460d98c6b0b8eea77169b83b3121cd963c6d7bad1" exitCode=0 Jul 24 15:36:34.405217 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:34.405063 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" event={"ID":"d7f7dfa0-21b2-44d9-b821-7d7af91d493c","Type":"ContainerDied","Data":"02ca6347db57f9cda3c832d460d98c6b0b8eea77169b83b3121cd963c6d7bad1"} Jul 24 15:36:35.244925 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:35.244893 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:35.245135 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:35.244893 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:35.245135 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:35.245018 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:35.245135 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:35.245027 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:35.245135 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:35.245102 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:35.245346 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:35.245183 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:35.410675 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:35.410640 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" event={"ID":"d7f7dfa0-21b2-44d9-b821-7d7af91d493c","Type":"ContainerStarted","Data":"bea0bdd33f10936a72754f4b822c6008175d0a2ef8495d589ae867c01a1cc228"} Jul 24 15:36:35.454323 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:35.454272 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-s7vsx" podStartSLOduration=8.008997584 podStartE2EDuration="36.454252527s" totalImagesPullingTime="28.445254943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:35.454084471 +0000 UTC m=+36.686812148" watchObservedRunningTime="2026-07-24 15:36:35.454252527 +0000 UTC m=+36.686980204" Jul 24 15:36:37.245243 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:37.245204 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:37.245766 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:37.245204 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:37.245766 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:37.245319 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-df4sc" podUID="cf25c7d5-8abf-40e0-bf77-9611072a1d43" Jul 24 15:36:37.245766 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:37.245419 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rbhv6" podUID="6a466ffd-d49f-45ea-ba0e-27bcdaf53f85" Jul 24 15:36:37.245766 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:37.245204 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:37.245766 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:37.245499 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4hnrl" podUID="b83152b7-d9db-42fc-8792-f6d38f731245" Jul 24 15:36:38.104077 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.104048 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-156.ec2.internal" event="NodeReady" Jul 24 15:36:38.104233 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.104163 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 15:36:38.138559 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.138526 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-br6wr"] Jul 24 15:36:38.157100 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.157072 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf"] Jul 24 15:36:38.157269 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.157202 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.160454 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.160027 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 15:36:38.160454 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.160129 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.160454 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.160289 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.160454 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.160453 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lbv2d\"" Jul 24 15:36:38.161484 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.161461 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 15:36:38.164645 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.164625 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 15:36:38.181008 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.180984 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68f5ff5d87-l2ht2"] Jul 24 15:36:38.181141 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.181099 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.183720 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.183670 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 15:36:38.183941 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.183917 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 15:36:38.184048 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.183953 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.184048 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.183964 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jp7hq\"" Jul 24 15:36:38.184048 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.184004 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.195097 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.195076 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29"] Jul 24 15:36:38.195217 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.195201 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.198276 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.198209 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 15:36:38.198276 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.198221 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 15:36:38.198468 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.198238 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ljdvh\"" Jul 24 15:36:38.200019 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.199792 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-67b59875d5-dh89s"] Jul 24 15:36:38.200604 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.200581 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:38.204469 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.204445 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 15:36:38.204647 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.204596 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx"] Jul 24 15:36:38.205900 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.205876 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.205999 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.205906 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.206091 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.206075 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.206492 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.206328 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 15:36:38.206829 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.206622 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-frq2x\"" Jul 24 15:36:38.209209 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.208162 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 15:36:38.209209 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.208380 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx"] Jul 24 15:36:38.209209 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.208481 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx" Jul 24 15:36:38.209209 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.208525 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.209209 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.208530 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 15:36:38.209209 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.208906 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6rbdl\"" Jul 24 15:36:38.209209 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.208940 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 15:36:38.209209 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.208909 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 15:36:38.209642 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.209252 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.209642 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.209420 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 15:36:38.210710 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.210672 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf"] Jul 24 15:36:38.210812 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.210735 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-br6wr"] Jul 24 15:36:38.210812 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.210757 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29"] Jul 24 15:36:38.210812 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.210783 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67b59875d5-dh89s"] Jul 24 15:36:38.210812 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.210790 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.210939 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.210833 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.210986 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.210839 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f5ff5d87-l2ht2"] Jul 24 15:36:38.211035 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.211017 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh"] Jul 24 15:36:38.211315 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.211299 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.211382 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.211347 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-x99db\"" Jul 24 15:36:38.212785 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.212765 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 15:36:38.213315 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.213300 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-x8299"] Jul 24 15:36:38.213463 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.213437 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.213463 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.213449 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.213587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.213465 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 15:36:38.213587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.213489 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.213587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.213501 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cq9ws\"" Jul 24 15:36:38.215210 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.215191 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xbcnh"] Jul 24 15:36:38.215307 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.215294 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.216108 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.216090 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 15:36:38.216335 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.216322 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-r2jwt\"" Jul 24 15:36:38.216556 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.216538 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.216678 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.216596 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 15:36:38.216678 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.216637 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.217314 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.217289 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q"] Jul 24 15:36:38.217412 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.217395 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.217523 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.217507 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-rcwvk\"" Jul 24 15:36:38.217799 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.217782 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 15:36:38.217926 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.217797 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 15:36:38.218049 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.218034 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.218162 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.218061 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.219238 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.219221 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw"] Jul 24 15:36:38.219374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.219360 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q" Jul 24 15:36:38.219538 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.219509 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 15:36:38.219913 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.219896 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 15:36:38.220113 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.220100 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6s28\"" Jul 24 15:36:38.221726 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.221707 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wkrdw\"" Jul 24 15:36:38.221807 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.221746 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.221944 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.221928 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dhtfb"] Jul 24 15:36:38.222011 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.222000 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:38.223504 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.223128 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.223504 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.223359 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 15:36:38.224326 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224304 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx"] Jul 24 15:36:38.224418 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224341 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx"] Jul 24 15:36:38.224418 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224370 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q"] Jul 24 15:36:38.224418 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224389 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh"] Jul 24 15:36:38.224418 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224412 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 15:36:38.224593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224415 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhtfb"] Jul 24 15:36:38.224593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224440 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 15:36:38.224593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224471 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhtfb" Jul 24 15:36:38.224593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224478 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbcnh"] Jul 24 15:36:38.224593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224512 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-x8299"] Jul 24 15:36:38.224593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224527 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9hqgw\"" Jul 24 15:36:38.224593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.224537 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw"] Jul 24 15:36:38.227087 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.227055 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lrd6q\"" Jul 24 15:36:38.227297 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.227278 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 15:36:38.227620 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.227600 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 15:36:38.230950 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.227604 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 15:36:38.306901 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.306865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b977c05-00e9-4a94-8c33-7c57e8b953e1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-scscf\" (UID: \"6b977c05-00e9-4a94-8c33-7c57e8b953e1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.306901 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.306900 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m2jd\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-kube-api-access-5m2jd\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.306916 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a726d78d-f872-44f8-ad72-1df1b141faba-config-volume\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.306934 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tg4j\" (UniqueName: \"kubernetes.io/projected/3b92d999-385e-4c9a-9f0e-6453933bacaf-kube-api-access-7tg4j\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.306982 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b977c05-00e9-4a94-8c33-7c57e8b953e1-config\") pod \"service-ca-operator-5d6c75f84f-scscf\" (UID: \"6b977c05-00e9-4a94-8c33-7c57e8b953e1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307013 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hnr\" (UniqueName: \"kubernetes.io/projected/4954048d-a95c-40db-ab9f-4dedccd0b5ab-kube-api-access-t4hnr\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307045 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-image-registry-private-configuration\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307151 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvrj7\" (UniqueName: \"kubernetes.io/projected/29328376-08a7-4a9d-a472-c78d6f3137fa-kube-api-access-lvrj7\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307208 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2a1411e-9fb7-422c-8946-169e281cdaa6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307234 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b92d999-385e-4c9a-9f0e-6453933bacaf-trusted-ca\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307255 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-certificates\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307277 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-installation-pull-secrets\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307302 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbee300-3ce7-4e7f-8c17-e56643547910-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307352 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c12ebe-3225-4061-953c-5dc160582a12-cert\") pod \"ingress-canary-dhtfb\" (UID: \"a0c12ebe-3225-4061-953c-5dc160582a12\") " pod="openshift-ingress-canary/ingress-canary-dhtfb" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307426 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b92d999-385e-4c9a-9f0e-6453933bacaf-serving-cert\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307459 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wks7f\" (UniqueName: \"kubernetes.io/projected/9fbee300-3ce7-4e7f-8c17-e56643547910-kube-api-access-wks7f\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.307587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307504 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-trusted-ca\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307537 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drrll\" (UniqueName: \"kubernetes.io/projected/7364f412-e2fc-43f2-88e4-c21f7da3ab54-kube-api-access-drrll\") pod \"volume-data-source-validator-549c6d54cf-ddvlx\" (UID: \"7364f412-e2fc-43f2-88e4-c21f7da3ab54\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307563 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx\" (UID: \"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307587 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307623 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307653 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9fbee300-3ce7-4e7f-8c17-e56643547910-snapshots\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307671 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbee300-3ce7-4e7f-8c17-e56643547910-service-ca-bundle\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307709 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/29328376-08a7-4a9d-a472-c78d6f3137fa-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307730 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a726d78d-f872-44f8-ad72-1df1b141faba-tmp-dir\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307745 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfdj\" (UniqueName: \"kubernetes.io/projected/a0c12ebe-3225-4061-953c-5dc160582a12-kube-api-access-9pfdj\") pod \"ingress-canary-dhtfb\" (UID: \"a0c12ebe-3225-4061-953c-5dc160582a12\") " pod="openshift-ingress-canary/ingress-canary-dhtfb" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307759 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fbee300-3ce7-4e7f-8c17-e56643547910-tmp\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307782 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307814 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww8zz\" (UniqueName: \"kubernetes.io/projected/2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a-kube-api-access-ww8zz\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx\" (UID: \"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307894 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-bound-sa-token\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307930 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9blk\" (UniqueName: \"kubernetes.io/projected/d16d9a0d-7deb-4a0f-8c6b-fb4dcce8f17d-kube-api-access-x9blk\") pod \"network-check-source-7d88cb55d8-x887q\" (UID: \"d16d9a0d-7deb-4a0f-8c6b-fb4dcce8f17d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q" Jul 24 15:36:38.308253 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.307977 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-default-certificate\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308007 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbpv\" (UniqueName: \"kubernetes.io/projected/9165b2df-4afe-4f54-b9e4-c16f0145e83e-kube-api-access-tgbpv\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308030 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308056 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx\" (UID: \"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308081 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtqcq\" (UniqueName: \"kubernetes.io/projected/6b977c05-00e9-4a94-8c33-7c57e8b953e1-kube-api-access-rtqcq\") pod \"service-ca-operator-5d6c75f84f-scscf\" (UID: \"6b977c05-00e9-4a94-8c33-7c57e8b953e1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308129 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzfwq\" (UniqueName: \"kubernetes.io/projected/a726d78d-f872-44f8-ad72-1df1b141faba-kube-api-access-lzfwq\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308157 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b92d999-385e-4c9a-9f0e-6453933bacaf-config\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308182 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fbee300-3ce7-4e7f-8c17-e56643547910-serving-cert\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308209 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308257 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308286 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308311 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-stats-auth\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.308780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.308337 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/189c7f88-3de8-4cce-872c-17ccf87b5885-ca-trust-extracted\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.409325 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b977c05-00e9-4a94-8c33-7c57e8b953e1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-scscf\" (UID: \"6b977c05-00e9-4a94-8c33-7c57e8b953e1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.409325 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409323 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m2jd\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-kube-api-access-5m2jd\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409342 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a726d78d-f872-44f8-ad72-1df1b141faba-config-volume\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409357 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tg4j\" (UniqueName: \"kubernetes.io/projected/3b92d999-385e-4c9a-9f0e-6453933bacaf-kube-api-access-7tg4j\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b977c05-00e9-4a94-8c33-7c57e8b953e1-config\") pod \"service-ca-operator-5d6c75f84f-scscf\" (UID: \"6b977c05-00e9-4a94-8c33-7c57e8b953e1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409403 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hnr\" (UniqueName: \"kubernetes.io/projected/4954048d-a95c-40db-ab9f-4dedccd0b5ab-kube-api-access-t4hnr\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409452 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-image-registry-private-configuration\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409494 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvrj7\" (UniqueName: \"kubernetes.io/projected/29328376-08a7-4a9d-a472-c78d6f3137fa-kube-api-access-lvrj7\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409518 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2a1411e-9fb7-422c-8946-169e281cdaa6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409540 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b92d999-385e-4c9a-9f0e-6453933bacaf-trusted-ca\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409563 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-certificates\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.409586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409587 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-installation-pull-secrets\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbee300-3ce7-4e7f-8c17-e56643547910-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409639 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c12ebe-3225-4061-953c-5dc160582a12-cert\") pod \"ingress-canary-dhtfb\" (UID: \"a0c12ebe-3225-4061-953c-5dc160582a12\") " pod="openshift-ingress-canary/ingress-canary-dhtfb" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b92d999-385e-4c9a-9f0e-6453933bacaf-serving-cert\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409688 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wks7f\" (UniqueName: \"kubernetes.io/projected/9fbee300-3ce7-4e7f-8c17-e56643547910-kube-api-access-wks7f\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409739 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-trusted-ca\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409772 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drrll\" (UniqueName: \"kubernetes.io/projected/7364f412-e2fc-43f2-88e4-c21f7da3ab54-kube-api-access-drrll\") pod \"volume-data-source-validator-549c6d54cf-ddvlx\" (UID: \"7364f412-e2fc-43f2-88e4-c21f7da3ab54\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409797 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx\" (UID: \"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409823 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409848 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409875 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9fbee300-3ce7-4e7f-8c17-e56643547910-snapshots\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbee300-3ce7-4e7f-8c17-e56643547910-service-ca-bundle\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409930 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/29328376-08a7-4a9d-a472-c78d6f3137fa-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409953 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a726d78d-f872-44f8-ad72-1df1b141faba-tmp-dir\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409976 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfdj\" (UniqueName: \"kubernetes.io/projected/a0c12ebe-3225-4061-953c-5dc160582a12-kube-api-access-9pfdj\") pod \"ingress-canary-dhtfb\" (UID: \"a0c12ebe-3225-4061-953c-5dc160582a12\") " pod="openshift-ingress-canary/ingress-canary-dhtfb" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.409998 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fbee300-3ce7-4e7f-8c17-e56643547910-tmp\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.410033 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410055 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww8zz\" (UniqueName: \"kubernetes.io/projected/2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a-kube-api-access-ww8zz\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx\" (UID: \"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410072 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a726d78d-f872-44f8-ad72-1df1b141faba-config-volume\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410081 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-bound-sa-token\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410105 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9blk\" (UniqueName: \"kubernetes.io/projected/d16d9a0d-7deb-4a0f-8c6b-fb4dcce8f17d-kube-api-access-x9blk\") pod \"network-check-source-7d88cb55d8-x887q\" (UID: \"d16d9a0d-7deb-4a0f-8c6b-fb4dcce8f17d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410146 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-default-certificate\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbpv\" (UniqueName: \"kubernetes.io/projected/9165b2df-4afe-4f54-b9e4-c16f0145e83e-kube-api-access-tgbpv\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410193 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410218 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx\" (UID: \"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410245 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtqcq\" (UniqueName: \"kubernetes.io/projected/6b977c05-00e9-4a94-8c33-7c57e8b953e1-kube-api-access-rtqcq\") pod \"service-ca-operator-5d6c75f84f-scscf\" (UID: \"6b977c05-00e9-4a94-8c33-7c57e8b953e1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410302 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzfwq\" (UniqueName: \"kubernetes.io/projected/a726d78d-f872-44f8-ad72-1df1b141faba-kube-api-access-lzfwq\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410330 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b92d999-385e-4c9a-9f0e-6453933bacaf-config\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410354 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fbee300-3ce7-4e7f-8c17-e56643547910-serving-cert\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410439 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410469 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410488 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-certificates\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.410680 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.410582 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:36:38.411316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410591 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b977c05-00e9-4a94-8c33-7c57e8b953e1-config\") pod \"service-ca-operator-5d6c75f84f-scscf\" (UID: \"6b977c05-00e9-4a94-8c33-7c57e8b953e1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.411316 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.410646 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls podName:4954048d-a95c-40db-ab9f-4dedccd0b5ab nodeName:}" failed. No retries permitted until 2026-07-24 15:36:38.910627786 +0000 UTC m=+40.143355445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls") pod "cluster-samples-operator-6748489656-94s29" (UID: "4954048d-a95c-40db-ab9f-4dedccd0b5ab") : secret "samples-operator-tls" not found Jul 24 15:36:38.411316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410997 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c2a1411e-9fb7-422c-8946-169e281cdaa6-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:38.411316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.410497 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:38.411316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.411057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-stats-auth\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.411316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.411087 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/189c7f88-3de8-4cce-872c-17ccf87b5885-ca-trust-extracted\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.411316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.411172 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbee300-3ce7-4e7f-8c17-e56643547910-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.411597 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.411415 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/189c7f88-3de8-4cce-872c-17ccf87b5885-ca-trust-extracted\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.411756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.411675 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-trusted-ca\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.412026 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.412003 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3b92d999-385e-4c9a-9f0e-6453933bacaf-trusted-ca\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.412324 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.412298 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/29328376-08a7-4a9d-a472-c78d6f3137fa-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.412569 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.412601 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a726d78d-f872-44f8-ad72-1df1b141faba-tmp-dir\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.412637 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert podName:c2a1411e-9fb7-422c-8946-169e281cdaa6 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:38.912620042 +0000 UTC m=+40.145347702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-kv8fw" (UID: "c2a1411e-9fb7-422c-8946-169e281cdaa6") : secret "networking-console-plugin-cert" not found Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.412726 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.412756 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fbee300-3ce7-4e7f-8c17-e56643547910-service-ca-bundle\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.412762 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls podName:a726d78d-f872-44f8-ad72-1df1b141faba nodeName:}" failed. No retries permitted until 2026-07-24 15:36:38.91275028 +0000 UTC m=+40.145477951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls") pod "dns-default-xbcnh" (UID: "a726d78d-f872-44f8-ad72-1df1b141faba") : secret "dns-default-metrics-tls" not found Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.412810 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fbee300-3ce7-4e7f-8c17-e56643547910-tmp\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.413003 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.413018 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f5ff5d87-l2ht2: secret "image-registry-tls" not found Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.413058 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls podName:189c7f88-3de8-4cce-872c-17ccf87b5885 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:38.913044578 +0000 UTC m=+40.145772238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls") pod "image-registry-68f5ff5d87-l2ht2" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885") : secret "image-registry-tls" not found Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.413104 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:36:38.413536 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.413161 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:38.913142513 +0000 UTC m=+40.145870183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : secret "router-metrics-certs-default" not found Jul 24 15:36:38.414226 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.413554 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b92d999-385e-4c9a-9f0e-6453933bacaf-config\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.414226 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.413672 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/9fbee300-3ce7-4e7f-8c17-e56643547910-snapshots\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.414226 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.414102 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx\" (UID: \"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.414226 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.414196 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:38.414385 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.414236 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:38.914217455 +0000 UTC m=+40.146945112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : configmap references non-existent config key: service-ca.crt Jul 24 15:36:38.415193 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.414848 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6b977c05-00e9-4a94-8c33-7c57e8b953e1-serving-cert\") pod \"service-ca-operator-5d6c75f84f-scscf\" (UID: \"6b977c05-00e9-4a94-8c33-7c57e8b953e1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.415193 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.414884 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3b92d999-385e-4c9a-9f0e-6453933bacaf-serving-cert\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.415193 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.414260 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls podName:29328376-08a7-4a9d-a472-c78d6f3137fa nodeName:}" failed. No retries permitted until 2026-07-24 15:36:38.914249379 +0000 UTC m=+40.146977040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pqnwh" (UID: "29328376-08a7-4a9d-a472-c78d6f3137fa") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:38.415193 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.415038 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-installation-pull-secrets\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.415721 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.415657 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-stats-auth\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.416227 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.416185 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9fbee300-3ce7-4e7f-8c17-e56643547910-serving-cert\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.416481 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.416459 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-image-registry-private-configuration\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.417142 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.417123 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0c12ebe-3225-4061-953c-5dc160582a12-cert\") pod \"ingress-canary-dhtfb\" (UID: \"a0c12ebe-3225-4061-953c-5dc160582a12\") " pod="openshift-ingress-canary/ingress-canary-dhtfb" Jul 24 15:36:38.417216 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.417196 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx\" (UID: \"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.417531 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.417513 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-default-certificate\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.426141 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.426108 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m2jd\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-kube-api-access-5m2jd\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.427059 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.426717 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wks7f\" (UniqueName: \"kubernetes.io/projected/9fbee300-3ce7-4e7f-8c17-e56643547910-kube-api-access-wks7f\") pod \"insights-operator-5885fdb759-br6wr\" (UID: \"9fbee300-3ce7-4e7f-8c17-e56643547910\") " pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.427165 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.427094 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tg4j\" (UniqueName: \"kubernetes.io/projected/3b92d999-385e-4c9a-9f0e-6453933bacaf-kube-api-access-7tg4j\") pod \"console-operator-584dcb6b54-x8299\" (UID: \"3b92d999-385e-4c9a-9f0e-6453933bacaf\") " pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.427501 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.427452 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-bound-sa-token\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.429119 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.428761 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9blk\" (UniqueName: \"kubernetes.io/projected/d16d9a0d-7deb-4a0f-8c6b-fb4dcce8f17d-kube-api-access-x9blk\") pod \"network-check-source-7d88cb55d8-x887q\" (UID: \"d16d9a0d-7deb-4a0f-8c6b-fb4dcce8f17d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q" Jul 24 15:36:38.429119 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.428782 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtqcq\" (UniqueName: \"kubernetes.io/projected/6b977c05-00e9-4a94-8c33-7c57e8b953e1-kube-api-access-rtqcq\") pod \"service-ca-operator-5d6c75f84f-scscf\" (UID: \"6b977c05-00e9-4a94-8c33-7c57e8b953e1\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.429119 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.429052 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hnr\" (UniqueName: \"kubernetes.io/projected/4954048d-a95c-40db-ab9f-4dedccd0b5ab-kube-api-access-t4hnr\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:38.429535 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.429491 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzfwq\" (UniqueName: \"kubernetes.io/projected/a726d78d-f872-44f8-ad72-1df1b141faba-kube-api-access-lzfwq\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.429535 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.429525 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvrj7\" (UniqueName: \"kubernetes.io/projected/29328376-08a7-4a9d-a472-c78d6f3137fa-kube-api-access-lvrj7\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.429875 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.429832 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbpv\" (UniqueName: \"kubernetes.io/projected/9165b2df-4afe-4f54-b9e4-c16f0145e83e-kube-api-access-tgbpv\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.430610 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.430589 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww8zz\" (UniqueName: \"kubernetes.io/projected/2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a-kube-api-access-ww8zz\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx\" (UID: \"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.430972 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.430947 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drrll\" (UniqueName: \"kubernetes.io/projected/7364f412-e2fc-43f2-88e4-c21f7da3ab54-kube-api-access-drrll\") pod \"volume-data-source-validator-549c6d54cf-ddvlx\" (UID: \"7364f412-e2fc-43f2-88e4-c21f7da3ab54\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx" Jul 24 15:36:38.431396 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.431375 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfdj\" (UniqueName: \"kubernetes.io/projected/a0c12ebe-3225-4061-953c-5dc160582a12-kube-api-access-9pfdj\") pod \"ingress-canary-dhtfb\" (UID: \"a0c12ebe-3225-4061-953c-5dc160582a12\") " pod="openshift-ingress-canary/ingress-canary-dhtfb" Jul 24 15:36:38.469073 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.468983 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-br6wr" Jul 24 15:36:38.493463 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.493436 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" Jul 24 15:36:38.531980 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.531948 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx" Jul 24 15:36:38.537883 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.537711 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" Jul 24 15:36:38.549570 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.549543 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:38.560567 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.560339 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q" Jul 24 15:36:38.572043 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.570933 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhtfb" Jul 24 15:36:38.688869 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.688047 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-br6wr"] Jul 24 15:36:38.705617 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.705590 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf"] Jul 24 15:36:38.782076 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.782044 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx"] Jul 24 15:36:38.785432 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.785197 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx"] Jul 24 15:36:38.788433 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:38.788396 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2044cfb4_b2d3_4b66_864e_df2d0ebc1b2a.slice/crio-36e8daf1a3d243e8411d0044cd675ab8858ce5ee90cfbbd5dbcba40f4d7a5e8e WatchSource:0}: Error finding container 36e8daf1a3d243e8411d0044cd675ab8858ce5ee90cfbbd5dbcba40f4d7a5e8e: Status 404 returned error can't find the container with id 36e8daf1a3d243e8411d0044cd675ab8858ce5ee90cfbbd5dbcba40f4d7a5e8e Jul 24 15:36:38.804852 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.804823 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-x8299"] Jul 24 15:36:38.805738 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:38.805714 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b92d999_385e_4c9a_9f0e_6453933bacaf.slice/crio-b7d80ea25b5056cf1a62acd0f1f8c35410b5c9ec2eed9b794b6a97ad4da50796 WatchSource:0}: Error finding container b7d80ea25b5056cf1a62acd0f1f8c35410b5c9ec2eed9b794b6a97ad4da50796: Status 404 returned error can't find the container with id b7d80ea25b5056cf1a62acd0f1f8c35410b5c9ec2eed9b794b6a97ad4da50796 Jul 24 15:36:38.808345 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.808323 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q"] Jul 24 15:36:38.810985 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:38.810961 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16d9a0d_7deb_4a0f_8c6b_fb4dcce8f17d.slice/crio-dd60b5c2319a03165a0743119e57a42c591207ad23c764a48382fcf9a22f976a WatchSource:0}: Error finding container dd60b5c2319a03165a0743119e57a42c591207ad23c764a48382fcf9a22f976a: Status 404 returned error can't find the container with id dd60b5c2319a03165a0743119e57a42c591207ad23c764a48382fcf9a22f976a Jul 24 15:36:38.813976 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.813957 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhtfb"] Jul 24 15:36:38.818044 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:36:38.818024 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c12ebe_3225_4061_953c_5dc160582a12.slice/crio-39177ee6e40a90a9ea49b3628f4be9037f5c74852875081ca2a5aeafc9bb97b2 WatchSource:0}: Error finding container 39177ee6e40a90a9ea49b3628f4be9037f5c74852875081ca2a5aeafc9bb97b2: Status 404 returned error can't find the container with id 39177ee6e40a90a9ea49b3628f4be9037f5c74852875081ca2a5aeafc9bb97b2 Jul 24 15:36:38.915913 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.915878 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:38.915913 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.915914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.916103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.915942 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:38.916103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.916002 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:38.916103 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916042 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:36:38.916103 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916048 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:36:38.916103 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916073 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f5ff5d87-l2ht2: secret "image-registry-tls" not found Jul 24 15:36:38.916103 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916098 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916105 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:39.916087555 +0000 UTC m=+41.148815216 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : secret "router-metrics-certs-default" not found Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916106 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916134 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls podName:189c7f88-3de8-4cce-872c-17ccf87b5885 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:39.916114862 +0000 UTC m=+41.148842537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls") pod "image-registry-68f5ff5d87-l2ht2" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885") : secret "image-registry-tls" not found Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.916158 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.916181 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:38.916203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916225 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls podName:a726d78d-f872-44f8-ad72-1df1b141faba nodeName:}" failed. No retries permitted until 2026-07-24 15:36:39.916211592 +0000 UTC m=+41.148939249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls") pod "dns-default-xbcnh" (UID: "a726d78d-f872-44f8-ad72-1df1b141faba") : secret "dns-default-metrics-tls" not found Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916260 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916248 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert podName:c2a1411e-9fb7-422c-8946-169e281cdaa6 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:39.916238241 +0000 UTC m=+41.148965899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-kv8fw" (UID: "c2a1411e-9fb7-422c-8946-169e281cdaa6") : secret "networking-console-plugin-cert" not found Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916272 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:38.916284 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916289 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:39.916279354 +0000 UTC m=+41.149007024 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : configmap references non-existent config key: service-ca.crt Jul 24 15:36:38.916605 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916305 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls podName:4954048d-a95c-40db-ab9f-4dedccd0b5ab nodeName:}" failed. No retries permitted until 2026-07-24 15:36:39.916297782 +0000 UTC m=+41.149025443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls") pod "cluster-samples-operator-6748489656-94s29" (UID: "4954048d-a95c-40db-ab9f-4dedccd0b5ab") : secret "samples-operator-tls" not found Jul 24 15:36:38.916605 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:38.916320 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls podName:29328376-08a7-4a9d-a472-c78d6f3137fa nodeName:}" failed. No retries permitted until 2026-07-24 15:36:39.916314416 +0000 UTC m=+41.149042071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pqnwh" (UID: "29328376-08a7-4a9d-a472-c78d6f3137fa") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:39.245859 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.245823 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:36:39.246055 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.245924 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:36:39.246055 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.245978 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:36:39.250270 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.250249 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:36:39.250390 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.250375 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t6g6g\"" Jul 24 15:36:39.250390 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.250382 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w77kg\"" Jul 24 15:36:39.251174 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.251154 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:36:39.431733 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.421682 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhtfb" event={"ID":"a0c12ebe-3225-4061-953c-5dc160582a12","Type":"ContainerStarted","Data":"39177ee6e40a90a9ea49b3628f4be9037f5c74852875081ca2a5aeafc9bb97b2"} Jul 24 15:36:39.431733 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.423185 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q" event={"ID":"d16d9a0d-7deb-4a0f-8c6b-fb4dcce8f17d","Type":"ContainerStarted","Data":"dd60b5c2319a03165a0743119e57a42c591207ad23c764a48382fcf9a22f976a"} Jul 24 15:36:39.431733 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.424547 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" event={"ID":"3b92d999-385e-4c9a-9f0e-6453933bacaf","Type":"ContainerStarted","Data":"b7d80ea25b5056cf1a62acd0f1f8c35410b5c9ec2eed9b794b6a97ad4da50796"} Jul 24 15:36:39.431733 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.425919 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx" event={"ID":"7364f412-e2fc-43f2-88e4-c21f7da3ab54","Type":"ContainerStarted","Data":"163cd017c05a0bf94201a559887e056ee67ded2a66d48c28672e57d2005637cc"} Jul 24 15:36:39.431733 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.427095 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" event={"ID":"6b977c05-00e9-4a94-8c33-7c57e8b953e1","Type":"ContainerStarted","Data":"266a009d9d68833e22214a4dd5c78f3b764b11a43b54404f0a4b085085a6ff59"} Jul 24 15:36:39.431733 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.428266 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" event={"ID":"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a","Type":"ContainerStarted","Data":"36e8daf1a3d243e8411d0044cd675ab8858ce5ee90cfbbd5dbcba40f4d7a5e8e"} Jul 24 15:36:39.431733 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.429508 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-br6wr" event={"ID":"9fbee300-3ce7-4e7f-8c17-e56643547910","Type":"ContainerStarted","Data":"315bae2fc42e9ab27ab3a97ed38b27d8e08496bc572b605bc6cea71a43bcea1e"} Jul 24 15:36:39.925122 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.925071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:39.925311 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.925159 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:39.925311 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.925226 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:39.925311 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.925251 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:39.925311 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.925280 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:39.925605 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.925365 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:39.925605 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:39.925391 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:39.925605 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.925529 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:36:39.925605 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.925596 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:41.925576182 +0000 UTC m=+43.158303840 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : secret "router-metrics-certs-default" not found Jul 24 15:36:39.926064 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926043 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:36:39.926131 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926108 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert podName:c2a1411e-9fb7-422c-8946-169e281cdaa6 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:41.926091311 +0000 UTC m=+43.158818985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-kv8fw" (UID: "c2a1411e-9fb7-422c-8946-169e281cdaa6") : secret "networking-console-plugin-cert" not found Jul 24 15:36:39.926192 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926170 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:36:39.926236 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926200 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls podName:a726d78d-f872-44f8-ad72-1df1b141faba nodeName:}" failed. No retries permitted until 2026-07-24 15:36:41.926189849 +0000 UTC m=+43.158917506 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls") pod "dns-default-xbcnh" (UID: "a726d78d-f872-44f8-ad72-1df1b141faba") : secret "dns-default-metrics-tls" not found Jul 24 15:36:39.926294 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926260 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:41.926251068 +0000 UTC m=+43.158978727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : configmap references non-existent config key: service-ca.crt Jul 24 15:36:39.926352 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926340 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:39.926398 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926373 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls podName:29328376-08a7-4a9d-a472-c78d6f3137fa nodeName:}" failed. No retries permitted until 2026-07-24 15:36:41.926362698 +0000 UTC m=+43.159090356 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pqnwh" (UID: "29328376-08a7-4a9d-a472-c78d6f3137fa") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:39.926464 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926421 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:36:39.926464 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926460 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls podName:4954048d-a95c-40db-ab9f-4dedccd0b5ab nodeName:}" failed. No retries permitted until 2026-07-24 15:36:41.926451407 +0000 UTC m=+43.159179080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls") pod "cluster-samples-operator-6748489656-94s29" (UID: "4954048d-a95c-40db-ab9f-4dedccd0b5ab") : secret "samples-operator-tls" not found Jul 24 15:36:39.926563 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926519 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:36:39.926563 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926530 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f5ff5d87-l2ht2: secret "image-registry-tls" not found Jul 24 15:36:39.926563 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:39.926557 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls podName:189c7f88-3de8-4cce-872c-17ccf87b5885 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:41.926548797 +0000 UTC m=+43.159276455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls") pod "image-registry-68f5ff5d87-l2ht2" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885") : secret "image-registry-tls" not found Jul 24 15:36:41.945088 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:41.944985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:41.945088 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:41.945070 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:41.945534 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:41.945091 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:41.945534 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:41.945117 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:41.945534 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:41.945209 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:41.945534 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:41.945235 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:41.945534 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:41.945272 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:41.945534 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.945414 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:36:41.945534 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.945480 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert podName:c2a1411e-9fb7-422c-8946-169e281cdaa6 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:45.945460148 +0000 UTC m=+47.178187810 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-kv8fw" (UID: "c2a1411e-9fb7-422c-8946-169e281cdaa6") : secret "networking-console-plugin-cert" not found Jul 24 15:36:41.945962 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.945941 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:36:41.946026 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946016 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls podName:a726d78d-f872-44f8-ad72-1df1b141faba nodeName:}" failed. No retries permitted until 2026-07-24 15:36:45.945999083 +0000 UTC m=+47.178726741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls") pod "dns-default-xbcnh" (UID: "a726d78d-f872-44f8-ad72-1df1b141faba") : secret "dns-default-metrics-tls" not found Jul 24 15:36:41.946096 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946084 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:45.9460738 +0000 UTC m=+47.178801457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : configmap references non-existent config key: service-ca.crt Jul 24 15:36:41.946160 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946148 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:41.946207 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946185 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls podName:29328376-08a7-4a9d-a472-c78d6f3137fa nodeName:}" failed. No retries permitted until 2026-07-24 15:36:45.946172617 +0000 UTC m=+47.178900278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pqnwh" (UID: "29328376-08a7-4a9d-a472-c78d6f3137fa") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:41.946252 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946239 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:36:41.946282 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946268 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls podName:4954048d-a95c-40db-ab9f-4dedccd0b5ab nodeName:}" failed. No retries permitted until 2026-07-24 15:36:45.946258842 +0000 UTC m=+47.178986514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls") pod "cluster-samples-operator-6748489656-94s29" (UID: "4954048d-a95c-40db-ab9f-4dedccd0b5ab") : secret "samples-operator-tls" not found Jul 24 15:36:41.946344 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946324 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:36:41.946344 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946341 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f5ff5d87-l2ht2: secret "image-registry-tls" not found Jul 24 15:36:41.946562 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946371 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls podName:189c7f88-3de8-4cce-872c-17ccf87b5885 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:45.946361319 +0000 UTC m=+47.179088990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls") pod "image-registry-68f5ff5d87-l2ht2" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885") : secret "image-registry-tls" not found Jul 24 15:36:41.946562 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946421 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:36:41.946562 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:41.946450 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:45.946441018 +0000 UTC m=+47.179168677 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : secret "router-metrics-certs-default" not found Jul 24 15:36:45.982908 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:45.982868 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:45.982908 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:45.982917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:45.982963 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:45.983006 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983046 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983070 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f5ff5d87-l2ht2: secret "image-registry-tls" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983115 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983125 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983125 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983125 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls podName:189c7f88-3de8-4cce-872c-17ccf87b5885 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:53.983105654 +0000 UTC m=+55.215833311 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls") pod "image-registry-68f5ff5d87-l2ht2" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885") : secret "image-registry-tls" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:45.983206 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:45.983241 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:45.983272 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983305 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls podName:a726d78d-f872-44f8-ad72-1df1b141faba nodeName:}" failed. No retries permitted until 2026-07-24 15:36:53.983285858 +0000 UTC m=+55.216013513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls") pod "dns-default-xbcnh" (UID: "a726d78d-f872-44f8-ad72-1df1b141faba") : secret "dns-default-metrics-tls" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983333 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:53.983324433 +0000 UTC m=+55.216052089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : secret "router-metrics-certs-default" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983339 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983347 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:45.983564 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983357 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert podName:c2a1411e-9fb7-422c-8946-169e281cdaa6 nodeName:}" failed. No retries permitted until 2026-07-24 15:36:53.983347778 +0000 UTC m=+55.216075439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-kv8fw" (UID: "c2a1411e-9fb7-422c-8946-169e281cdaa6") : secret "networking-console-plugin-cert" not found Jul 24 15:36:45.984240 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983374 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:36:53.983367244 +0000 UTC m=+55.216094900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : configmap references non-existent config key: service-ca.crt Jul 24 15:36:45.984240 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983400 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls podName:4954048d-a95c-40db-ab9f-4dedccd0b5ab nodeName:}" failed. No retries permitted until 2026-07-24 15:36:53.98338335 +0000 UTC m=+55.216111012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls") pod "cluster-samples-operator-6748489656-94s29" (UID: "4954048d-a95c-40db-ab9f-4dedccd0b5ab") : secret "samples-operator-tls" not found Jul 24 15:36:45.984240 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:45.983420 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls podName:29328376-08a7-4a9d-a472-c78d6f3137fa nodeName:}" failed. No retries permitted until 2026-07-24 15:36:53.983410142 +0000 UTC m=+55.216137803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pqnwh" (UID: "29328376-08a7-4a9d-a472-c78d6f3137fa") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:47.457157 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.456908 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx" event={"ID":"7364f412-e2fc-43f2-88e4-c21f7da3ab54","Type":"ContainerStarted","Data":"6ea2363c290a023e8a8d2775209d3339382372435f49f633d4c2e56a27f066c7"} Jul 24 15:36:47.458713 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.458667 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" event={"ID":"6b977c05-00e9-4a94-8c33-7c57e8b953e1","Type":"ContainerStarted","Data":"6a0f6d33fef875004fc6fff9502f61d02714c9f555f09307ba8174f132464dc0"} Jul 24 15:36:47.460743 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.460717 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" event={"ID":"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a","Type":"ContainerStarted","Data":"fd89369d3f1650988dacc6b59ca416782d76fcd46943bdfd4e15d7a3ff28c1db"} Jul 24 15:36:47.462673 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.462645 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-br6wr" event={"ID":"9fbee300-3ce7-4e7f-8c17-e56643547910","Type":"ContainerStarted","Data":"15370fe0a70a8d684b0cfdca9426af2ac4944e0acf01c3075da61552021041ca"} Jul 24 15:36:47.464367 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.464334 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhtfb" event={"ID":"a0c12ebe-3225-4061-953c-5dc160582a12","Type":"ContainerStarted","Data":"4ad45d8fdf113ceb04aef76850d851f66488407aa289828673e3b646cfa6b502"} Jul 24 15:36:47.465972 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.465948 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q" event={"ID":"d16d9a0d-7deb-4a0f-8c6b-fb4dcce8f17d","Type":"ContainerStarted","Data":"ec21a9f30bc91a086dcc90288011c26ecc188ee5cacb80dd36d43acca61e06ac"} Jul 24 15:36:47.467820 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.467803 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/0.log" Jul 24 15:36:47.467920 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.467835 2571 generic.go:356] "Generic (PLEG): container finished" podID="3b92d999-385e-4c9a-9f0e-6453933bacaf" containerID="65ce57442d7ac3dc58ae91c15723d519eeee20c10373af1ef90e420b991b85f0" exitCode=255 Jul 24 15:36:47.467920 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.467860 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" event={"ID":"3b92d999-385e-4c9a-9f0e-6453933bacaf","Type":"ContainerDied","Data":"65ce57442d7ac3dc58ae91c15723d519eeee20c10373af1ef90e420b991b85f0"} Jul 24 15:36:47.468095 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.468078 2571 scope.go:117] "RemoveContainer" containerID="65ce57442d7ac3dc58ae91c15723d519eeee20c10373af1ef90e420b991b85f0" Jul 24 15:36:47.475580 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.475538 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddvlx" podStartSLOduration=32.054237427 podStartE2EDuration="38.475524592s" totalImagesPullingTime="6.421287165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:47.47497089 +0000 UTC m=+48.707698569" watchObservedRunningTime="2026-07-24 15:36:47.475524592 +0000 UTC m=+48.708252269" Jul 24 15:36:47.491974 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.491573 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" podStartSLOduration=29.705298856 podStartE2EDuration="37.491559173s" totalImagesPullingTime="7.786260317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:47.491222609 +0000 UTC m=+48.723950300" watchObservedRunningTime="2026-07-24 15:36:47.491559173 +0000 UTC m=+48.724286854" Jul 24 15:36:47.510967 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.510919 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dhtfb" podStartSLOduration=1.7430211930000001 podStartE2EDuration="9.510901514s" totalImagesPullingTime="7.767880321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:47.509808775 +0000 UTC m=+48.742536455" watchObservedRunningTime="2026-07-24 15:36:47.510901514 +0000 UTC m=+48.743629196" Jul 24 15:36:47.533947 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.533893 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" podStartSLOduration=30.691789235 podStartE2EDuration="38.533850183s" totalImagesPullingTime="7.842060948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:47.533620393 +0000 UTC m=+48.766348072" watchObservedRunningTime="2026-07-24 15:36:47.533850183 +0000 UTC m=+48.766577854" Jul 24 15:36:47.576053 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.575997 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-x887q" podStartSLOduration=29.811604825 podStartE2EDuration="37.575973346s" totalImagesPullingTime="7.764368521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:47.575203615 +0000 UTC m=+48.807931294" watchObservedRunningTime="2026-07-24 15:36:47.575973346 +0000 UTC m=+48.808701024" Jul 24 15:36:47.595730 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:47.595539 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-br6wr" podStartSLOduration=30.747928456 podStartE2EDuration="38.595515885s" totalImagesPullingTime="7.847587429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:47.594030031 +0000 UTC m=+48.826757727" watchObservedRunningTime="2026-07-24 15:36:47.595515885 +0000 UTC m=+48.828243564" Jul 24 15:36:48.474599 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.474508 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/1.log" Jul 24 15:36:48.475460 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.475071 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/0.log" Jul 24 15:36:48.475460 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.475122 2571 generic.go:356] "Generic (PLEG): container finished" podID="3b92d999-385e-4c9a-9f0e-6453933bacaf" containerID="20e3267e98f96aba3712cfe9695e1f441a9eab7a19d1aed8692bd7c747fa0ee3" exitCode=255 Jul 24 15:36:48.475460 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.475223 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" event={"ID":"3b92d999-385e-4c9a-9f0e-6453933bacaf","Type":"ContainerDied","Data":"20e3267e98f96aba3712cfe9695e1f441a9eab7a19d1aed8692bd7c747fa0ee3"} Jul 24 15:36:48.475460 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.475319 2571 scope.go:117] "RemoveContainer" containerID="65ce57442d7ac3dc58ae91c15723d519eeee20c10373af1ef90e420b991b85f0" Jul 24 15:36:48.475784 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.475630 2571 scope.go:117] "RemoveContainer" containerID="20e3267e98f96aba3712cfe9695e1f441a9eab7a19d1aed8692bd7c747fa0ee3" Jul 24 15:36:48.476038 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:48.475845 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-x8299_openshift-console-operator(3b92d999-385e-4c9a-9f0e-6453933bacaf)\"" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" podUID="3b92d999-385e-4c9a-9f0e-6453933bacaf" Jul 24 15:36:48.503017 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.502990 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k8m5d"] Jul 24 15:36:48.519609 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.519587 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k8m5d"] Jul 24 15:36:48.519772 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.519717 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.522213 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.522190 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pmnt\"" Jul 24 15:36:48.522464 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.522449 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 15:36:48.522613 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.522594 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 15:36:48.550802 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.550770 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:48.550949 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.550830 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:36:48.615572 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.615538 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5920135f-0896-42f3-9851-5ee72589bc8c-crio-socket\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.615572 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.615572 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.615783 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.615669 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5920135f-0896-42f3-9851-5ee72589bc8c-data-volume\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.615821 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.615791 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4h6x\" (UniqueName: \"kubernetes.io/projected/5920135f-0896-42f3-9851-5ee72589bc8c-kube-api-access-q4h6x\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.615854 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.615824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.717074 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.717033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4h6x\" (UniqueName: \"kubernetes.io/projected/5920135f-0896-42f3-9851-5ee72589bc8c-kube-api-access-q4h6x\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.717245 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.717088 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.717245 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.717109 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5920135f-0896-42f3-9851-5ee72589bc8c-crio-socket\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.717245 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.717128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.717245 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:48.717192 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:36:48.717245 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.717217 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5920135f-0896-42f3-9851-5ee72589bc8c-data-volume\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.717245 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.717220 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5920135f-0896-42f3-9851-5ee72589bc8c-crio-socket\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.717446 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:48.717260 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:36:49.217236963 +0000 UTC m=+50.449964619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : configmap "kube-rbac-proxy" not found Jul 24 15:36:48.717446 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:48.717292 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:36:48.717446 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:48.717343 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:36:49.217327262 +0000 UTC m=+50.450054917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : secret "insights-runtime-extractor-tls" not found Jul 24 15:36:48.717560 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.717535 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5920135f-0896-42f3-9851-5ee72589bc8c-data-volume\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:48.727964 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:48.727900 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4h6x\" (UniqueName: \"kubernetes.io/projected/5920135f-0896-42f3-9851-5ee72589bc8c-kube-api-access-q4h6x\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:49.221845 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:49.221805 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:49.222030 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:49.221895 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:49.222030 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:49.221930 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:36:49.222030 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:49.222016 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:36:50.221995534 +0000 UTC m=+51.454723209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : configmap "kube-rbac-proxy" not found Jul 24 15:36:49.222030 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:49.222026 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:36:49.222190 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:49.222066 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:36:50.222054526 +0000 UTC m=+51.454782201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : secret "insights-runtime-extractor-tls" not found Jul 24 15:36:49.483396 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:49.483324 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/1.log" Jul 24 15:36:49.483772 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:49.483675 2571 scope.go:117] "RemoveContainer" containerID="20e3267e98f96aba3712cfe9695e1f441a9eab7a19d1aed8692bd7c747fa0ee3" Jul 24 15:36:49.483895 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:49.483858 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-x8299_openshift-console-operator(3b92d999-385e-4c9a-9f0e-6453933bacaf)\"" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" podUID="3b92d999-385e-4c9a-9f0e-6453933bacaf" Jul 24 15:36:50.231599 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:50.231561 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:50.231822 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:50.231612 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:50.231822 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:50.231717 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:36:50.231822 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:50.231785 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:36:50.231822 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:50.231795 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:36:52.231775544 +0000 UTC m=+53.464503223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : configmap "kube-rbac-proxy" not found Jul 24 15:36:50.231822 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:50.231821 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:36:52.231810742 +0000 UTC m=+53.464538397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : secret "insights-runtime-extractor-tls" not found Jul 24 15:36:50.486467 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:50.486390 2571 scope.go:117] "RemoveContainer" containerID="20e3267e98f96aba3712cfe9695e1f441a9eab7a19d1aed8692bd7c747fa0ee3" Jul 24 15:36:50.486842 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:50.486544 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-x8299_openshift-console-operator(3b92d999-385e-4c9a-9f0e-6453933bacaf)\"" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" podUID="3b92d999-385e-4c9a-9f0e-6453933bacaf" Jul 24 15:36:50.890625 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:50.890597 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pl8t9_b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9/dns-node-resolver/0.log" Jul 24 15:36:51.888856 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:51.888821 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b9smv_da43f1b5-a039-43c2-be38-894330b7745e/node-ca/0.log" Jul 24 15:36:52.249149 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:52.249114 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:52.249307 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:52.249171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:52.249307 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:52.249244 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:36:52.249376 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:52.249318 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:36:56.249299392 +0000 UTC m=+57.482027072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : configmap "kube-rbac-proxy" not found Jul 24 15:36:52.249376 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:52.249344 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:36:52.249448 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:52.249397 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:36:56.249384799 +0000 UTC m=+57.482112455 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : secret "insights-runtime-extractor-tls" not found Jul 24 15:36:52.488615 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:52.488584 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dhtfb_a0c12ebe-3225-4061-953c-5dc160582a12/serve-healthcheck-canary/0.log" Jul 24 15:36:53.288664 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:53.288634 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx_2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a/kube-storage-version-migrator-operator/0.log" Jul 24 15:36:54.065570 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:54.065532 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:36:54.065570 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:54.065572 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:54.065597 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:54.065683 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065720 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065729 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065745 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68f5ff5d87-l2ht2: secret "image-registry-tls" not found Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065783 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert podName:c2a1411e-9fb7-422c-8946-169e281cdaa6 nodeName:}" failed. No retries permitted until 2026-07-24 15:37:10.06576454 +0000 UTC m=+71.298492219 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-kv8fw" (UID: "c2a1411e-9fb7-422c-8946-169e281cdaa6") : secret "networking-console-plugin-cert" not found Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065793 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065803 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls podName:189c7f88-3de8-4cce-872c-17ccf87b5885 nodeName:}" failed. No retries permitted until 2026-07-24 15:37:10.065792513 +0000 UTC m=+71.298520172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls") pod "image-registry-68f5ff5d87-l2ht2" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885") : secret "image-registry-tls" not found Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065799 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065849 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls podName:a726d78d-f872-44f8-ad72-1df1b141faba nodeName:}" failed. No retries permitted until 2026-07-24 15:37:10.065832807 +0000 UTC m=+71.298560465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls") pod "dns-default-xbcnh" (UID: "a726d78d-f872-44f8-ad72-1df1b141faba") : secret "dns-default-metrics-tls" not found Jul 24 15:36:54.065869 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:54.065872 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:36:54.066283 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:54.065903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:36:54.066283 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065918 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:37:10.065907878 +0000 UTC m=+71.298635534 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : secret "router-metrics-certs-default" not found Jul 24 15:36:54.066283 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:54.065960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:36:54.066283 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065966 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:54.066283 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.065972 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle podName:9165b2df-4afe-4f54-b9e4-c16f0145e83e nodeName:}" failed. No retries permitted until 2026-07-24 15:37:10.065961453 +0000 UTC m=+71.298689114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle") pod "router-default-67b59875d5-dh89s" (UID: "9165b2df-4afe-4f54-b9e4-c16f0145e83e") : configmap references non-existent config key: service-ca.crt Jul 24 15:36:54.066283 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.066001 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls podName:29328376-08a7-4a9d-a472-c78d6f3137fa nodeName:}" failed. No retries permitted until 2026-07-24 15:37:10.065991479 +0000 UTC m=+71.298719136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-pqnwh" (UID: "29328376-08a7-4a9d-a472-c78d6f3137fa") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:36:54.066283 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.066043 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 15:36:54.066283 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:54.066077 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls podName:4954048d-a95c-40db-ab9f-4dedccd0b5ab nodeName:}" failed. No retries permitted until 2026-07-24 15:37:10.066068796 +0000 UTC m=+71.298796472 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls") pod "cluster-samples-operator-6748489656-94s29" (UID: "4954048d-a95c-40db-ab9f-4dedccd0b5ab") : secret "samples-operator-tls" not found Jul 24 15:36:56.287163 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:56.287127 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:56.287561 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:36:56.287171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:36:56.287561 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:56.287273 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:36:56.287561 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:56.287277 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:36:56.287561 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:56.287346 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:37:04.287329127 +0000 UTC m=+65.520056785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : configmap "kube-rbac-proxy" not found Jul 24 15:36:56.287561 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:36:56.287389 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:37:04.287371261 +0000 UTC m=+65.520098919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : secret "insights-runtime-extractor-tls" not found Jul 24 15:37:03.410439 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:03.410404 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czgtt" Jul 24 15:37:04.360901 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:04.360846 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:37:04.360901 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:04.360903 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:37:04.361125 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:37:04.361088 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:37:04.361166 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:37:04.361157 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls podName:5920135f-0896-42f3-9851-5ee72589bc8c nodeName:}" failed. No retries permitted until 2026-07-24 15:37:20.361138954 +0000 UTC m=+81.593866610 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8m5d" (UID: "5920135f-0896-42f3-9851-5ee72589bc8c") : secret "insights-runtime-extractor-tls" not found Jul 24 15:37:04.361360 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:04.361339 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5920135f-0896-42f3-9851-5ee72589bc8c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:37:04.866336 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:04.866309 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:37:04.869305 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:04.869287 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:37:04.878924 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:04.878894 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b83152b7-d9db-42fc-8792-f6d38f731245-metrics-certs\") pod \"network-metrics-daemon-4hnrl\" (UID: \"b83152b7-d9db-42fc-8792-f6d38f731245\") " pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:37:04.967065 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:04.967024 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:37:04.970555 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:04.970536 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:37:04.980218 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:04.980198 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6a466ffd-d49f-45ea-ba0e-27bcdaf53f85-original-pull-secret\") pod \"global-pull-secret-syncer-rbhv6\" (UID: \"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85\") " pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:37:05.057303 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.057271 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rbhv6" Jul 24 15:37:05.068170 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.068138 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:37:05.070646 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.070622 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w77kg\"" Jul 24 15:37:05.070909 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.070888 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2fdg\" (UniqueName: \"kubernetes.io/projected/cf25c7d5-8abf-40e0-bf77-9611072a1d43-kube-api-access-t2fdg\") pod \"network-check-target-df4sc\" (UID: \"cf25c7d5-8abf-40e0-bf77-9611072a1d43\") " pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:37:05.078105 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.078060 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4hnrl" Jul 24 15:37:05.201794 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.201758 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rbhv6"] Jul 24 15:37:05.205530 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:05.205499 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a466ffd_d49f_45ea_ba0e_27bcdaf53f85.slice/crio-1798dd31d5ff5efbc412d349fc4bf179a8f9dbd6e10100e6381146c563184b51 WatchSource:0}: Error finding container 1798dd31d5ff5efbc412d349fc4bf179a8f9dbd6e10100e6381146c563184b51: Status 404 returned error can't find the container with id 1798dd31d5ff5efbc412d349fc4bf179a8f9dbd6e10100e6381146c563184b51 Jul 24 15:37:05.220763 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.220737 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4hnrl"] Jul 24 15:37:05.222965 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:05.222942 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83152b7_d9db_42fc_8792_f6d38f731245.slice/crio-3f3f19db9a4404f511f069c018af37d4ad733cf82c9dec07148b4b865eb2adcd WatchSource:0}: Error finding container 3f3f19db9a4404f511f069c018af37d4ad733cf82c9dec07148b4b865eb2adcd: Status 404 returned error can't find the container with id 3f3f19db9a4404f511f069c018af37d4ad733cf82c9dec07148b4b865eb2adcd Jul 24 15:37:05.245204 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.245181 2571 scope.go:117] "RemoveContainer" containerID="20e3267e98f96aba3712cfe9695e1f441a9eab7a19d1aed8692bd7c747fa0ee3" Jul 24 15:37:05.366112 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.366091 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-t6g6g\"" Jul 24 15:37:05.374052 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.374026 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:37:05.520092 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.520061 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-df4sc"] Jul 24 15:37:05.522849 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:05.522823 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf25c7d5_8abf_40e0_bf77_9611072a1d43.slice/crio-a6a402c20076cfd35eb90a7aecd2fe7fd04b7f65fab61acfb3a46445eb73fa68 WatchSource:0}: Error finding container a6a402c20076cfd35eb90a7aecd2fe7fd04b7f65fab61acfb3a46445eb73fa68: Status 404 returned error can't find the container with id a6a402c20076cfd35eb90a7aecd2fe7fd04b7f65fab61acfb3a46445eb73fa68 Jul 24 15:37:05.532533 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.532499 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-df4sc" event={"ID":"cf25c7d5-8abf-40e0-bf77-9611072a1d43","Type":"ContainerStarted","Data":"a6a402c20076cfd35eb90a7aecd2fe7fd04b7f65fab61acfb3a46445eb73fa68"} Jul 24 15:37:05.533532 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.533509 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4hnrl" event={"ID":"b83152b7-d9db-42fc-8792-f6d38f731245","Type":"ContainerStarted","Data":"3f3f19db9a4404f511f069c018af37d4ad733cf82c9dec07148b4b865eb2adcd"} Jul 24 15:37:05.534874 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.534859 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:37:05.535271 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.535256 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/1.log" Jul 24 15:37:05.535316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.535287 2571 generic.go:356] "Generic (PLEG): container finished" podID="3b92d999-385e-4c9a-9f0e-6453933bacaf" containerID="29a02196a29a72db58a9664dd55130573390060203a534b222a29476afc853cf" exitCode=255 Jul 24 15:37:05.535379 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.535361 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" event={"ID":"3b92d999-385e-4c9a-9f0e-6453933bacaf","Type":"ContainerDied","Data":"29a02196a29a72db58a9664dd55130573390060203a534b222a29476afc853cf"} Jul 24 15:37:05.535427 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.535407 2571 scope.go:117] "RemoveContainer" containerID="20e3267e98f96aba3712cfe9695e1f441a9eab7a19d1aed8692bd7c747fa0ee3" Jul 24 15:37:05.535801 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.535783 2571 scope.go:117] "RemoveContainer" containerID="29a02196a29a72db58a9664dd55130573390060203a534b222a29476afc853cf" Jul 24 15:37:05.536115 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:37:05.536094 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-x8299_openshift-console-operator(3b92d999-385e-4c9a-9f0e-6453933bacaf)\"" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" podUID="3b92d999-385e-4c9a-9f0e-6453933bacaf" Jul 24 15:37:05.537237 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:05.537210 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rbhv6" event={"ID":"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85","Type":"ContainerStarted","Data":"1798dd31d5ff5efbc412d349fc4bf179a8f9dbd6e10100e6381146c563184b51"} Jul 24 15:37:06.544762 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:06.544718 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-df4sc" event={"ID":"cf25c7d5-8abf-40e0-bf77-9611072a1d43","Type":"ContainerStarted","Data":"fddb51911bc62864781138ede7e0845bf0b5086a0593aac82efdecce0aec5095"} Jul 24 15:37:06.545505 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:06.545480 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:37:06.547485 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:06.547459 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:37:06.563778 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:06.563722 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-df4sc" podStartSLOduration=67.563681944 podStartE2EDuration="1m7.563681944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:06.562666834 +0000 UTC m=+67.795394515" watchObservedRunningTime="2026-07-24 15:37:06.563681944 +0000 UTC m=+67.796409625" Jul 24 15:37:07.552774 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:07.552734 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4hnrl" event={"ID":"b83152b7-d9db-42fc-8792-f6d38f731245","Type":"ContainerStarted","Data":"ec9e2abec8f6b9c5c638ecf6aec3e34ad35fb299f8f61b4162859d2a65cba9b4"} Jul 24 15:37:07.553262 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:07.552797 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4hnrl" event={"ID":"b83152b7-d9db-42fc-8792-f6d38f731245","Type":"ContainerStarted","Data":"3dda23bae5b7b835f87c7161789a92c941035a35b1d54b4c7b949e4cd441264c"} Jul 24 15:37:07.574159 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:07.574109 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4hnrl" podStartSLOduration=66.923434384 podStartE2EDuration="1m8.574092504s" totalImagesPullingTime="1.65065812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:07.572800582 +0000 UTC m=+68.805528261" watchObservedRunningTime="2026-07-24 15:37:07.574092504 +0000 UTC m=+68.806820182" Jul 24 15:37:08.550675 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:08.550632 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:37:08.550887 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:08.550707 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:37:08.551099 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:08.551080 2571 scope.go:117] "RemoveContainer" containerID="29a02196a29a72db58a9664dd55130573390060203a534b222a29476afc853cf" Jul 24 15:37:08.551300 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:37:08.551280 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-x8299_openshift-console-operator(3b92d999-385e-4c9a-9f0e-6453933bacaf)\"" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" podUID="3b92d999-385e-4c9a-9f0e-6453933bacaf" Jul 24 15:37:10.118561 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.118494 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:37:10.118561 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.118552 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:37:10.118561 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.118578 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:37:10.119193 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.118651 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:37:10.119193 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.118741 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:37:10.119193 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.118892 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:37:10.119193 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.118940 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:37:10.119618 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.119571 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9165b2df-4afe-4f54-b9e4-c16f0145e83e-service-ca-bundle\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:37:10.121297 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.121276 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9165b2df-4afe-4f54-b9e4-c16f0145e83e-metrics-certs\") pod \"router-default-67b59875d5-dh89s\" (UID: \"9165b2df-4afe-4f54-b9e4-c16f0145e83e\") " pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:37:10.121410 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.121355 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a726d78d-f872-44f8-ad72-1df1b141faba-metrics-tls\") pod \"dns-default-xbcnh\" (UID: \"a726d78d-f872-44f8-ad72-1df1b141faba\") " pod="openshift-dns/dns-default-xbcnh" Jul 24 15:37:10.121797 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.121768 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c2a1411e-9fb7-422c-8946-169e281cdaa6-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-kv8fw\" (UID: \"c2a1411e-9fb7-422c-8946-169e281cdaa6\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:37:10.121892 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.121859 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4954048d-a95c-40db-ab9f-4dedccd0b5ab-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-94s29\" (UID: \"4954048d-a95c-40db-ab9f-4dedccd0b5ab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:37:10.121933 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.121896 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"image-registry-68f5ff5d87-l2ht2\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:37:10.121965 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.121952 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/29328376-08a7-4a9d-a472-c78d6f3137fa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-pqnwh\" (UID: \"29328376-08a7-4a9d-a472-c78d6f3137fa\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:37:10.313440 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.313407 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ljdvh\"" Jul 24 15:37:10.320705 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.320678 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:37:10.320909 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.320894 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-frq2x\"" Jul 24 15:37:10.327369 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.327350 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-6rbdl\"" Jul 24 15:37:10.328416 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.328392 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" Jul 24 15:37:10.335025 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.335001 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:37:10.346288 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.346268 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-r2jwt\"" Jul 24 15:37:10.353686 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.353664 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" Jul 24 15:37:10.358531 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.358506 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6s28\"" Jul 24 15:37:10.366725 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.366499 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xbcnh" Jul 24 15:37:10.369793 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.369666 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9hqgw\"" Jul 24 15:37:10.378251 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.378025 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" Jul 24 15:37:10.525196 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.525162 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68f5ff5d87-l2ht2"] Jul 24 15:37:10.529288 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:10.529242 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod189c7f88_3de8_4cce_872c_17ccf87b5885.slice/crio-a93b2d955a271021ad958bb74272156e7d34468a93b5cd1633e002d682977897 WatchSource:0}: Error finding container a93b2d955a271021ad958bb74272156e7d34468a93b5cd1633e002d682977897: Status 404 returned error can't find the container with id a93b2d955a271021ad958bb74272156e7d34468a93b5cd1633e002d682977897 Jul 24 15:37:10.548814 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.548786 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29"] Jul 24 15:37:10.549811 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.549790 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67b59875d5-dh89s"] Jul 24 15:37:10.567775 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.567741 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67b59875d5-dh89s" event={"ID":"9165b2df-4afe-4f54-b9e4-c16f0145e83e","Type":"ContainerStarted","Data":"9c6c132dd362d02cc20b830ca4216c63d842d77a9f8ab786a568e2e08c5cc385"} Jul 24 15:37:10.569677 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.569637 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rbhv6" event={"ID":"6a466ffd-d49f-45ea-ba0e-27bcdaf53f85","Type":"ContainerStarted","Data":"7dfd2cf5bc9984512b630b540176cad838106c1a50357144a14516288c315cc3"} Jul 24 15:37:10.571623 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.571507 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" event={"ID":"189c7f88-3de8-4cce-872c-17ccf87b5885","Type":"ContainerStarted","Data":"a93b2d955a271021ad958bb74272156e7d34468a93b5cd1633e002d682977897"} Jul 24 15:37:10.582412 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.582137 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh"] Jul 24 15:37:10.582412 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:10.582382 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29328376_08a7_4a9d_a472_c78d6f3137fa.slice/crio-ffa7fc02d5aa890e3de13c2b94f4f8c1ebeecad8571bdad9f86a8b661bb9cc9e WatchSource:0}: Error finding container ffa7fc02d5aa890e3de13c2b94f4f8c1ebeecad8571bdad9f86a8b661bb9cc9e: Status 404 returned error can't find the container with id ffa7fc02d5aa890e3de13c2b94f4f8c1ebeecad8571bdad9f86a8b661bb9cc9e Jul 24 15:37:10.593574 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.593532 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rbhv6" podStartSLOduration=67.277577937 podStartE2EDuration="1m11.593518084s" totalImagesPullingTime="4.315940147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:10.592961924 +0000 UTC m=+71.825689615" watchObservedRunningTime="2026-07-24 15:37:10.593518084 +0000 UTC m=+71.826245752" Jul 24 15:37:10.606202 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.606160 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw"] Jul 24 15:37:10.608591 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:10.608571 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xbcnh"] Jul 24 15:37:10.608800 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:10.608771 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a1411e_9fb7_422c_8946_169e281cdaa6.slice/crio-a788946d4acca35783a53c5cfde488f7a4532bbc9aa3f6fe6966f367cabf9052 WatchSource:0}: Error finding container a788946d4acca35783a53c5cfde488f7a4532bbc9aa3f6fe6966f367cabf9052: Status 404 returned error can't find the container with id a788946d4acca35783a53c5cfde488f7a4532bbc9aa3f6fe6966f367cabf9052 Jul 24 15:37:11.579970 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:11.579896 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" event={"ID":"189c7f88-3de8-4cce-872c-17ccf87b5885","Type":"ContainerStarted","Data":"af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04"} Jul 24 15:37:11.580435 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:11.580179 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:37:11.582707 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:11.582612 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" event={"ID":"c2a1411e-9fb7-422c-8946-169e281cdaa6","Type":"ContainerStarted","Data":"a788946d4acca35783a53c5cfde488f7a4532bbc9aa3f6fe6966f367cabf9052"} Jul 24 15:37:11.584262 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:11.584232 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbcnh" event={"ID":"a726d78d-f872-44f8-ad72-1df1b141faba","Type":"ContainerStarted","Data":"ada01f0cb3edb9f6365236216d75f5c7197e1249780b0bafdc535bae4b7b261b"} Jul 24 15:37:11.586726 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:11.586681 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67b59875d5-dh89s" event={"ID":"9165b2df-4afe-4f54-b9e4-c16f0145e83e","Type":"ContainerStarted","Data":"9452ed2251cc571335de8257d2e7b48e7dff0cb8393888f9a63d4675ca1409ba"} Jul 24 15:37:11.588044 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:11.587996 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" event={"ID":"29328376-08a7-4a9d-a472-c78d6f3137fa","Type":"ContainerStarted","Data":"ffa7fc02d5aa890e3de13c2b94f4f8c1ebeecad8571bdad9f86a8b661bb9cc9e"} Jul 24 15:37:11.590151 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:11.590128 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" event={"ID":"4954048d-a95c-40db-ab9f-4dedccd0b5ab","Type":"ContainerStarted","Data":"97a899be4f191956ab333553971952dbdd33896be6122033f85c96e4458e136e"} Jul 24 15:37:11.603376 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:11.603324 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" podStartSLOduration=72.603306108 podStartE2EDuration="1m12.603306108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:11.602005249 +0000 UTC m=+72.834732984" watchObservedRunningTime="2026-07-24 15:37:11.603306108 +0000 UTC m=+72.836033787" Jul 24 15:37:11.622424 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:11.622374 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-67b59875d5-dh89s" podStartSLOduration=62.622354867 podStartE2EDuration="1m2.622354867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:11.621470489 +0000 UTC m=+72.854198207" watchObservedRunningTime="2026-07-24 15:37:11.622354867 +0000 UTC m=+72.855082549" Jul 24 15:37:12.335623 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:12.335579 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:37:12.338516 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:12.338487 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:37:12.593640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:12.593560 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:37:12.594729 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:12.594685 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-67b59875d5-dh89s" Jul 24 15:37:14.082048 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.082020 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g"] Jul 24 15:37:14.096591 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.096563 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g"] Jul 24 15:37:14.096758 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.096708 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" Jul 24 15:37:14.100329 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.100265 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 15:37:14.101774 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.101752 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7sjkg\"" Jul 24 15:37:14.101904 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.101788 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 15:37:14.101904 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.101856 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 15:37:14.102162 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.102148 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 15:37:14.256632 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.256582 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/39533059-7348-492a-b2e1-17f49c57f496-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65c6b6dffb-lf77g\" (UID: \"39533059-7348-492a-b2e1-17f49c57f496\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" Jul 24 15:37:14.256734 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.256637 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbn5k\" (UniqueName: \"kubernetes.io/projected/39533059-7348-492a-b2e1-17f49c57f496-kube-api-access-bbn5k\") pod \"managed-serviceaccount-addon-agent-65c6b6dffb-lf77g\" (UID: \"39533059-7348-492a-b2e1-17f49c57f496\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" Jul 24 15:37:14.357084 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.357035 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbn5k\" (UniqueName: \"kubernetes.io/projected/39533059-7348-492a-b2e1-17f49c57f496-kube-api-access-bbn5k\") pod \"managed-serviceaccount-addon-agent-65c6b6dffb-lf77g\" (UID: \"39533059-7348-492a-b2e1-17f49c57f496\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" Jul 24 15:37:14.357159 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.357127 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/39533059-7348-492a-b2e1-17f49c57f496-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65c6b6dffb-lf77g\" (UID: \"39533059-7348-492a-b2e1-17f49c57f496\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" Jul 24 15:37:14.363001 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.362398 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/39533059-7348-492a-b2e1-17f49c57f496-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-65c6b6dffb-lf77g\" (UID: \"39533059-7348-492a-b2e1-17f49c57f496\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" Jul 24 15:37:14.368519 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.368441 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbn5k\" (UniqueName: \"kubernetes.io/projected/39533059-7348-492a-b2e1-17f49c57f496-kube-api-access-bbn5k\") pod \"managed-serviceaccount-addon-agent-65c6b6dffb-lf77g\" (UID: \"39533059-7348-492a-b2e1-17f49c57f496\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" Jul 24 15:37:14.416449 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.416427 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" Jul 24 15:37:14.601902 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.601865 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" event={"ID":"c2a1411e-9fb7-422c-8946-169e281cdaa6","Type":"ContainerStarted","Data":"09ef0b904d90d6fa924faa681270b59a95c1c4284131a857a43d9ad45c42d235"} Jul 24 15:37:14.603776 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.603718 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbcnh" event={"ID":"a726d78d-f872-44f8-ad72-1df1b141faba","Type":"ContainerStarted","Data":"37291af024db2cf66b0a824d8edef7f4dccbc9590698b37018c17297480f4bf0"} Jul 24 15:37:14.605286 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.605263 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" event={"ID":"29328376-08a7-4a9d-a472-c78d6f3137fa","Type":"ContainerStarted","Data":"21470c36be09e287c388796fade20352f28f174f1f9f374d63191d66c7aa0f5a"} Jul 24 15:37:14.608186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.608155 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" event={"ID":"4954048d-a95c-40db-ab9f-4dedccd0b5ab","Type":"ContainerStarted","Data":"a3b7e966c1c9575c029b40e4c2561f5e4f1ff3155bbe930f1f353b01b8e5d8c2"} Jul 24 15:37:14.608307 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.608203 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" event={"ID":"4954048d-a95c-40db-ab9f-4dedccd0b5ab","Type":"ContainerStarted","Data":"370618b4d9be27dd28022481718b1220f6ce7b1da4e641a4a73a5dd80fb2c038"} Jul 24 15:37:14.610356 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.610330 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g"] Jul 24 15:37:14.612853 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:14.612833 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39533059_7348_492a_b2e1_17f49c57f496.slice/crio-b0ab42decf82e84d0ad1052d19e6ed222d89fc7885977dbff631f21633ab97f8 WatchSource:0}: Error finding container b0ab42decf82e84d0ad1052d19e6ed222d89fc7885977dbff631f21633ab97f8: Status 404 returned error can't find the container with id b0ab42decf82e84d0ad1052d19e6ed222d89fc7885977dbff631f21633ab97f8 Jul 24 15:37:14.627155 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.626809 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-kv8fw" podStartSLOduration=57.988169068 podStartE2EDuration="1m1.626795571s" totalImagesPullingTime="3.638626503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:14.625556039 +0000 UTC m=+75.858283711" watchObservedRunningTime="2026-07-24 15:37:14.626795571 +0000 UTC m=+75.859523284" Jul 24 15:37:14.649117 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.649027 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-pqnwh" podStartSLOduration=60.97791065 podStartE2EDuration="1m4.649007762s" totalImagesPullingTime="3.671097112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:14.647894227 +0000 UTC m=+75.880621907" watchObservedRunningTime="2026-07-24 15:37:14.649007762 +0000 UTC m=+75.881735438" Jul 24 15:37:14.675928 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:14.675868 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-94s29" podStartSLOduration=62.052151786 podStartE2EDuration="1m5.675845752s" totalImagesPullingTime="3.623693966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:14.675100651 +0000 UTC m=+75.907828324" watchObservedRunningTime="2026-07-24 15:37:14.675845752 +0000 UTC m=+75.908573431" Jul 24 15:37:15.612627 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:15.612590 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xbcnh" event={"ID":"a726d78d-f872-44f8-ad72-1df1b141faba","Type":"ContainerStarted","Data":"6e362f82e06c67ab958f9a19c8851d8a17125953a8b3f9a7fc23b12c445c75f3"} Jul 24 15:37:15.613122 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:15.612739 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xbcnh" Jul 24 15:37:15.613945 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:15.613917 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" event={"ID":"39533059-7348-492a-b2e1-17f49c57f496","Type":"ContainerStarted","Data":"b0ab42decf82e84d0ad1052d19e6ed222d89fc7885977dbff631f21633ab97f8"} Jul 24 15:37:15.630871 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:15.630824 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xbcnh" podStartSLOduration=33.994730745 podStartE2EDuration="37.630807752s" totalImagesPullingTime="3.636077007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:15.629478817 +0000 UTC m=+76.862206510" watchObservedRunningTime="2026-07-24 15:37:15.630807752 +0000 UTC m=+76.863535429" Jul 24 15:37:18.627542 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:18.627502 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" event={"ID":"39533059-7348-492a-b2e1-17f49c57f496","Type":"ContainerStarted","Data":"130a73e3a7b1a5de057e82e022add813cfe9a6452f58824cfab6a9e07d061794"} Jul 24 15:37:18.645743 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:18.645673 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-65c6b6dffb-lf77g" podStartSLOduration=1.427433663 podStartE2EDuration="4.64565421s" totalImagesPullingTime="3.218220547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:18.644648989 +0000 UTC m=+79.877376693" watchObservedRunningTime="2026-07-24 15:37:18.64565421 +0000 UTC m=+79.878381888" Jul 24 15:37:20.409062 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:20.408940 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:37:20.411280 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:20.411258 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5920135f-0896-42f3-9851-5ee72589bc8c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8m5d\" (UID: \"5920135f-0896-42f3-9851-5ee72589bc8c\") " pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:37:20.631102 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:20.631069 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pmnt\"" Jul 24 15:37:20.639431 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:20.639407 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k8m5d" Jul 24 15:37:20.770719 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:20.770676 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k8m5d"] Jul 24 15:37:20.773074 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:20.773047 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5920135f_0896_42f3_9851_5ee72589bc8c.slice/crio-32db45e6f4b3de73fbe2f61b558de3f30b5b4c3a626d6ac71c21df88584d3d52 WatchSource:0}: Error finding container 32db45e6f4b3de73fbe2f61b558de3f30b5b4c3a626d6ac71c21df88584d3d52: Status 404 returned error can't find the container with id 32db45e6f4b3de73fbe2f61b558de3f30b5b4c3a626d6ac71c21df88584d3d52 Jul 24 15:37:21.248226 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:21.248190 2571 scope.go:117] "RemoveContainer" containerID="29a02196a29a72db58a9664dd55130573390060203a534b222a29476afc853cf" Jul 24 15:37:21.248450 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:37:21.248400 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-x8299_openshift-console-operator(3b92d999-385e-4c9a-9f0e-6453933bacaf)\"" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" podUID="3b92d999-385e-4c9a-9f0e-6453933bacaf" Jul 24 15:37:21.640794 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:21.640631 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k8m5d" event={"ID":"5920135f-0896-42f3-9851-5ee72589bc8c","Type":"ContainerStarted","Data":"32db45e6f4b3de73fbe2f61b558de3f30b5b4c3a626d6ac71c21df88584d3d52"} Jul 24 15:37:25.620367 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:25.620344 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xbcnh" Jul 24 15:37:25.659238 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:25.659202 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k8m5d" event={"ID":"5920135f-0896-42f3-9851-5ee72589bc8c","Type":"ContainerStarted","Data":"fff05ec9feebe7f55b5bf427d769722c209795fd2bb3d0b277de9b6a89c8e789"} Jul 24 15:37:27.354743 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.354682 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m75qf"] Jul 24 15:37:27.362055 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.362033 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.364814 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.364792 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 15:37:27.364941 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.364817 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 15:37:27.365890 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.365874 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9cbmn\"" Jul 24 15:37:27.365994 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.365878 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 15:37:27.366496 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.366481 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 15:37:27.471239 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.471210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6554e04-268b-415a-a050-13cd4a345939-metrics-client-ca\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.471239 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.471244 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-textfile\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.471495 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.471272 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.471495 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.471301 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-accelerators-collector-config\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.471495 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.471379 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-tls\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.471495 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.471414 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6554e04-268b-415a-a050-13cd4a345939-sys\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.471495 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.471444 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6554e04-268b-415a-a050-13cd4a345939-root\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.471495 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.471473 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-wtmp\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.471775 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.471506 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j7nz\" (UniqueName: \"kubernetes.io/projected/e6554e04-268b-415a-a050-13cd4a345939-kube-api-access-8j7nz\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.572747 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572686 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-tls\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.572905 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6554e04-268b-415a-a050-13cd4a345939-sys\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.572905 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572784 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6554e04-268b-415a-a050-13cd4a345939-root\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.572905 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-wtmp\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.572905 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572839 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j7nz\" (UniqueName: \"kubernetes.io/projected/e6554e04-268b-415a-a050-13cd4a345939-kube-api-access-8j7nz\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.572905 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572891 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6554e04-268b-415a-a050-13cd4a345939-sys\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.573110 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572910 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6554e04-268b-415a-a050-13cd4a345939-metrics-client-ca\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.573110 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572895 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6554e04-268b-415a-a050-13cd4a345939-root\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.573110 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572937 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-textfile\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.573110 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.573110 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572985 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-wtmp\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.573110 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.572999 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-accelerators-collector-config\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.573405 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.573334 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-textfile\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.573579 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.573554 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6554e04-268b-415a-a050-13cd4a345939-metrics-client-ca\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.573940 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.573916 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-accelerators-collector-config\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.575577 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.575554 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-tls\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.575685 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.575581 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6554e04-268b-415a-a050-13cd4a345939-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.581087 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.581066 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j7nz\" (UniqueName: \"kubernetes.io/projected/e6554e04-268b-415a-a050-13cd4a345939-kube-api-access-8j7nz\") pod \"node-exporter-m75qf\" (UID: \"e6554e04-268b-415a-a050-13cd4a345939\") " pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:27.674607 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:27.674528 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m75qf" Jul 24 15:37:29.030342 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:29.030296 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6554e04_268b_415a_a050_13cd4a345939.slice/crio-ecb3f55c4f5d4990b7e0ef76102c91ac03d5950cb89f68678d8ef4ba82772491 WatchSource:0}: Error finding container ecb3f55c4f5d4990b7e0ef76102c91ac03d5950cb89f68678d8ef4ba82772491: Status 404 returned error can't find the container with id ecb3f55c4f5d4990b7e0ef76102c91ac03d5950cb89f68678d8ef4ba82772491 Jul 24 15:37:29.680527 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:29.680486 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k8m5d" event={"ID":"5920135f-0896-42f3-9851-5ee72589bc8c","Type":"ContainerStarted","Data":"e85ec4c34dc898394a8165c58f188647d3b4409ae8ce701519107ad9e0662da4"} Jul 24 15:37:29.682172 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:29.681865 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m75qf" event={"ID":"e6554e04-268b-415a-a050-13cd4a345939","Type":"ContainerStarted","Data":"ecb3f55c4f5d4990b7e0ef76102c91ac03d5950cb89f68678d8ef4ba82772491"} Jul 24 15:37:30.688460 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:30.688428 2571 generic.go:356] "Generic (PLEG): container finished" podID="e6554e04-268b-415a-a050-13cd4a345939" containerID="ee40aedf2a7444caf1ef22d5b618769d32f8b0c0efb14c5bc98a91859ced3f94" exitCode=0 Jul 24 15:37:30.688932 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:30.688503 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m75qf" event={"ID":"e6554e04-268b-415a-a050-13cd4a345939","Type":"ContainerDied","Data":"ee40aedf2a7444caf1ef22d5b618769d32f8b0c0efb14c5bc98a91859ced3f94"} Jul 24 15:37:31.695674 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:31.695625 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m75qf" event={"ID":"e6554e04-268b-415a-a050-13cd4a345939","Type":"ContainerStarted","Data":"b683fa25ea9deb8c937d0135bacc70446f5834a1167f482d469acbb85581d440"} Jul 24 15:37:31.696113 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:31.695723 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m75qf" event={"ID":"e6554e04-268b-415a-a050-13cd4a345939","Type":"ContainerStarted","Data":"9586d852a224fd1a094ecbf8d21dd86281b436ffaf008ae9e2b43e6b8f003654"} Jul 24 15:37:31.719050 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:31.719001 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m75qf" podStartSLOduration=3.596023631 podStartE2EDuration="4.718980792s" totalImagesPullingTime="1.040977944s" totalInitContainerRuntime="81.979217ms" isStatefulPod=true podCreationTimestamp="2026-07-24 15:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:31.718982009 +0000 UTC m=+92.951709687" watchObservedRunningTime="2026-07-24 15:37:31.718980792 +0000 UTC m=+92.951708471" Jul 24 15:37:32.598238 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:32.598211 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:37:33.245216 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:33.245183 2571 scope.go:117] "RemoveContainer" containerID="29a02196a29a72db58a9664dd55130573390060203a534b222a29476afc853cf" Jul 24 15:37:33.710183 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:33.710111 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:37:33.710331 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:33.710229 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" event={"ID":"3b92d999-385e-4c9a-9f0e-6453933bacaf","Type":"ContainerStarted","Data":"dfde2a1afec903cede1ca155082a3a1dfe822a66c02952cbeccc1b8760119624"} Jul 24 15:37:33.710565 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:33.710538 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:37:33.712495 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:33.712469 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k8m5d" event={"ID":"5920135f-0896-42f3-9851-5ee72589bc8c","Type":"ContainerStarted","Data":"f9e7cb11b8979eccc85fd9bfd800dcd1b39c323c31c5202b0b31b67331a011c4"} Jul 24 15:37:33.734308 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:33.734267 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" podStartSLOduration=75.963656914 podStartE2EDuration="1m23.734254533s" totalImagesPullingTime="7.770597619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:33.732761826 +0000 UTC m=+94.965489509" watchObservedRunningTime="2026-07-24 15:37:33.734254533 +0000 UTC m=+94.966982211" Jul 24 15:37:33.752975 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:33.752930 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k8m5d" podStartSLOduration=33.910436385 podStartE2EDuration="45.752916681s" totalImagesPullingTime="11.842480296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:36:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:33.751652262 +0000 UTC m=+94.984379941" watchObservedRunningTime="2026-07-24 15:37:33.752916681 +0000 UTC m=+94.985644360" Jul 24 15:37:34.270177 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.270149 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-x8299" Jul 24 15:37:34.441453 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.441419 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-9b5h5"] Jul 24 15:37:34.443545 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.443531 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-9b5h5" Jul 24 15:37:34.446110 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.446086 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 15:37:34.446336 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.446321 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 15:37:34.446390 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.446381 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-krwmb\"" Jul 24 15:37:34.453438 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.453418 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-9b5h5"] Jul 24 15:37:34.525934 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.525866 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fxh\" (UniqueName: \"kubernetes.io/projected/c730a2cb-903c-486c-bd2b-d7b63013c18a-kube-api-access-m5fxh\") pod \"downloads-7c97dc4555-9b5h5\" (UID: \"c730a2cb-903c-486c-bd2b-d7b63013c18a\") " pod="openshift-console/downloads-7c97dc4555-9b5h5" Jul 24 15:37:34.626993 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.626960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fxh\" (UniqueName: \"kubernetes.io/projected/c730a2cb-903c-486c-bd2b-d7b63013c18a-kube-api-access-m5fxh\") pod \"downloads-7c97dc4555-9b5h5\" (UID: \"c730a2cb-903c-486c-bd2b-d7b63013c18a\") " pod="openshift-console/downloads-7c97dc4555-9b5h5" Jul 24 15:37:34.635524 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.635489 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fxh\" (UniqueName: \"kubernetes.io/projected/c730a2cb-903c-486c-bd2b-d7b63013c18a-kube-api-access-m5fxh\") pod \"downloads-7c97dc4555-9b5h5\" (UID: \"c730a2cb-903c-486c-bd2b-d7b63013c18a\") " pod="openshift-console/downloads-7c97dc4555-9b5h5" Jul 24 15:37:34.753118 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.753086 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-9b5h5" Jul 24 15:37:34.886056 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:34.886031 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-9b5h5"] Jul 24 15:37:34.888515 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:34.888476 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc730a2cb_903c_486c_bd2b_d7b63013c18a.slice/crio-bdd3324107fca18738ab12a9c55b3988f9216b44339c364079255a670bf6d5f1 WatchSource:0}: Error finding container bdd3324107fca18738ab12a9c55b3988f9216b44339c364079255a670bf6d5f1: Status 404 returned error can't find the container with id bdd3324107fca18738ab12a9c55b3988f9216b44339c364079255a670bf6d5f1 Jul 24 15:37:35.723863 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:35.723823 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-9b5h5" event={"ID":"c730a2cb-903c-486c-bd2b-d7b63013c18a","Type":"ContainerStarted","Data":"bdd3324107fca18738ab12a9c55b3988f9216b44339c364079255a670bf6d5f1"} Jul 24 15:37:36.452330 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:36.452293 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f5ff5d87-l2ht2"] Jul 24 15:37:38.559728 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:38.559679 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-df4sc" Jul 24 15:37:40.486499 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.486469 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d66b9dbc8-5l4sp"] Jul 24 15:37:40.516970 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.516937 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d66b9dbc8-5l4sp"] Jul 24 15:37:40.517147 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.517075 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.520086 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.520061 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 15:37:40.520231 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.520061 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2r65f\"" Jul 24 15:37:40.520231 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.520101 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 15:37:40.520231 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.520108 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 15:37:40.521285 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.521258 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 15:37:40.521391 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.521296 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 15:37:40.580387 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.580351 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-service-ca\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.580583 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.580396 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-oauth-serving-cert\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.580583 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.580445 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbp4\" (UniqueName: \"kubernetes.io/projected/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-kube-api-access-wtbp4\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.580583 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.580527 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-config\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.580583 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.580564 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-serving-cert\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.580795 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.580620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-oauth-config\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.681463 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.681417 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-oauth-config\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.681643 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.681508 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-service-ca\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.681643 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.681539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-oauth-serving-cert\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.681643 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.681579 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbp4\" (UniqueName: \"kubernetes.io/projected/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-kube-api-access-wtbp4\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.681643 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.681609 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-config\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.681643 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.681636 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-serving-cert\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.683232 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.683177 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-config\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.683406 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.683380 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-oauth-serving-cert\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.683590 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.683569 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-service-ca\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.684901 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.684877 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-serving-cert\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.685020 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.684992 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-oauth-config\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.690605 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.690581 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbp4\" (UniqueName: \"kubernetes.io/projected/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-kube-api-access-wtbp4\") pod \"console-d66b9dbc8-5l4sp\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:40.828744 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:40.828711 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:41.002339 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:41.002309 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d66b9dbc8-5l4sp"] Jul 24 15:37:41.005582 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:41.005553 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f7e2a6_cfaa_4f0a_935a_fc7b30fc466f.slice/crio-1b0a16467af4197ddeb4dc12086111f49f5379908f0e336f07b3b7638c7bc60c WatchSource:0}: Error finding container 1b0a16467af4197ddeb4dc12086111f49f5379908f0e336f07b3b7638c7bc60c: Status 404 returned error can't find the container with id 1b0a16467af4197ddeb4dc12086111f49f5379908f0e336f07b3b7638c7bc60c Jul 24 15:37:41.753402 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:41.753361 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d66b9dbc8-5l4sp" event={"ID":"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f","Type":"ContainerStarted","Data":"1b0a16467af4197ddeb4dc12086111f49f5379908f0e336f07b3b7638c7bc60c"} Jul 24 15:37:44.769651 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:44.769613 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d66b9dbc8-5l4sp" event={"ID":"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f","Type":"ContainerStarted","Data":"e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3"} Jul 24 15:37:44.788741 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:44.788675 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d66b9dbc8-5l4sp" podStartSLOduration=1.327239354 podStartE2EDuration="4.788657836s" totalImagesPullingTime="3.461418482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:37:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:44.787050558 +0000 UTC m=+106.019778237" watchObservedRunningTime="2026-07-24 15:37:44.788657836 +0000 UTC m=+106.021385514" Jul 24 15:37:49.946103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:49.946072 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fdf49f974-fw5fg"] Jul 24 15:37:49.983459 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:49.983420 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdf49f974-fw5fg"] Jul 24 15:37:49.983622 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:49.983553 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:49.991683 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:49.991652 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 15:37:50.069390 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.069355 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-oauth-config\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.069587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.069438 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-serving-cert\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.069587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.069466 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-config\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.069587 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.069500 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-oauth-serving-cert\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.069744 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.069613 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4dfs\" (UniqueName: \"kubernetes.io/projected/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-kube-api-access-h4dfs\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.069744 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.069707 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-service-ca\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.069825 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.069765 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-trusted-ca-bundle\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.170569 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.170519 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4dfs\" (UniqueName: \"kubernetes.io/projected/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-kube-api-access-h4dfs\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.170785 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.170588 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-service-ca\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.170785 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.170619 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-trusted-ca-bundle\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.170785 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.170646 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-oauth-config\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.170785 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.170725 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-serving-cert\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.170785 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.170746 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-config\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.171252 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.171221 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-oauth-serving-cert\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.171500 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.171452 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-config\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.171500 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.171458 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-service-ca\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.171795 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.171762 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-trusted-ca-bundle\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.171894 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.171878 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-oauth-serving-cert\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.174096 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.174077 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-serving-cert\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.174200 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.174129 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-oauth-config\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.179263 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.179242 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4dfs\" (UniqueName: \"kubernetes.io/projected/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-kube-api-access-h4dfs\") pod \"console-5fdf49f974-fw5fg\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.295189 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.295157 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:37:50.829336 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.829306 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:50.829537 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.829413 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:50.835279 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:50.835255 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:51.806414 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:51.806386 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:37:51.975113 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:51.975085 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fdf49f974-fw5fg"] Jul 24 15:37:51.977842 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:37:51.977812 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16bddfc_50b5_4ad8_9ecf_dcb77fa7acd1.slice/crio-c902967aba133f7ec0a53f9b3dcf30bd20074250ceeb5bcbb99239bd61c85b6f WatchSource:0}: Error finding container c902967aba133f7ec0a53f9b3dcf30bd20074250ceeb5bcbb99239bd61c85b6f: Status 404 returned error can't find the container with id c902967aba133f7ec0a53f9b3dcf30bd20074250ceeb5bcbb99239bd61c85b6f Jul 24 15:37:52.810004 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:52.809949 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdf49f974-fw5fg" event={"ID":"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1","Type":"ContainerStarted","Data":"03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa"} Jul 24 15:37:52.810465 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:52.810021 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdf49f974-fw5fg" event={"ID":"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1","Type":"ContainerStarted","Data":"c902967aba133f7ec0a53f9b3dcf30bd20074250ceeb5bcbb99239bd61c85b6f"} Jul 24 15:37:52.811865 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:52.811835 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-9b5h5" event={"ID":"c730a2cb-903c-486c-bd2b-d7b63013c18a","Type":"ContainerStarted","Data":"fea6b0bba8e39398441d4d665d9d8d7874e7e964594b0763ef5b427b736ef3a1"} Jul 24 15:37:52.812210 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:52.812182 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-9b5h5" Jul 24 15:37:52.828239 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:52.828196 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fdf49f974-fw5fg" podStartSLOduration=3.828182296 podStartE2EDuration="3.828182296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:52.826491905 +0000 UTC m=+114.059219585" watchObservedRunningTime="2026-07-24 15:37:52.828182296 +0000 UTC m=+114.060909975" Jul 24 15:37:52.831140 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:52.831115 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-9b5h5" Jul 24 15:37:52.843108 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:37:52.843059 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-9b5h5" podStartSLOduration=1.599610666 podStartE2EDuration="18.843043831s" totalImagesPullingTime="17.243433165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:37:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:52.84141658 +0000 UTC m=+114.074144263" watchObservedRunningTime="2026-07-24 15:37:52.843043831 +0000 UTC m=+114.075771508" Jul 24 15:38:00.295487 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:00.295447 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:38:00.296039 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:00.295553 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:38:00.301164 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:00.301136 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:38:00.858390 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:00.858354 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:38:00.912498 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:00.912452 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d66b9dbc8-5l4sp"] Jul 24 15:38:01.478477 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.478403 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" podUID="189c7f88-3de8-4cce-872c-17ccf87b5885" containerName="registry" containerID="cri-o://af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04" gracePeriod=30 Jul 24 15:38:01.769046 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.769015 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:38:01.859133 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.859101 2571 generic.go:356] "Generic (PLEG): container finished" podID="189c7f88-3de8-4cce-872c-17ccf87b5885" containerID="af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04" exitCode=0 Jul 24 15:38:01.859295 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.859147 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" Jul 24 15:38:01.859295 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.859187 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" event={"ID":"189c7f88-3de8-4cce-872c-17ccf87b5885","Type":"ContainerDied","Data":"af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04"} Jul 24 15:38:01.859295 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.859242 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68f5ff5d87-l2ht2" event={"ID":"189c7f88-3de8-4cce-872c-17ccf87b5885","Type":"ContainerDied","Data":"a93b2d955a271021ad958bb74272156e7d34468a93b5cd1633e002d682977897"} Jul 24 15:38:01.859295 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.859271 2571 scope.go:117] "RemoveContainer" containerID="af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04" Jul 24 15:38:01.867446 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.867424 2571 scope.go:117] "RemoveContainer" containerID="af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04" Jul 24 15:38:01.867759 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:38:01.867729 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04\": container with ID starting with af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04 not found: ID does not exist" containerID="af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04" Jul 24 15:38:01.867840 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.867772 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04"} err="failed to get container status \"af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04\": rpc error: code = NotFound desc = could not find container \"af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04\": container with ID starting with af5f7cf461cc983cf60500d271a52f057ed6a4ce35a43e104fbb407ac6b90e04 not found: ID does not exist" Jul 24 15:38:01.880952 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.880927 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5m2jd\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-kube-api-access-5m2jd\") pod \"189c7f88-3de8-4cce-872c-17ccf87b5885\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " Jul 24 15:38:01.881096 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.880960 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-image-registry-private-configuration\") pod \"189c7f88-3de8-4cce-872c-17ccf87b5885\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " Jul 24 15:38:01.881096 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.880989 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") pod \"189c7f88-3de8-4cce-872c-17ccf87b5885\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " Jul 24 15:38:01.881096 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.881023 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-certificates\") pod \"189c7f88-3de8-4cce-872c-17ccf87b5885\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " Jul 24 15:38:01.881096 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.881051 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-installation-pull-secrets\") pod \"189c7f88-3de8-4cce-872c-17ccf87b5885\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " Jul 24 15:38:01.881096 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.881077 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-trusted-ca\") pod \"189c7f88-3de8-4cce-872c-17ccf87b5885\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " Jul 24 15:38:01.881332 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.881112 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/189c7f88-3de8-4cce-872c-17ccf87b5885-ca-trust-extracted\") pod \"189c7f88-3de8-4cce-872c-17ccf87b5885\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " Jul 24 15:38:01.881332 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.881169 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-bound-sa-token\") pod \"189c7f88-3de8-4cce-872c-17ccf87b5885\" (UID: \"189c7f88-3de8-4cce-872c-17ccf87b5885\") " Jul 24 15:38:01.881923 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.881626 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "189c7f88-3de8-4cce-872c-17ccf87b5885" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:38:01.881923 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.881764 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "189c7f88-3de8-4cce-872c-17ccf87b5885" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:38:01.883643 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.883617 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-kube-api-access-5m2jd" (OuterVolumeSpecName: "kube-api-access-5m2jd") pod "189c7f88-3de8-4cce-872c-17ccf87b5885" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885"). InnerVolumeSpecName "kube-api-access-5m2jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:38:01.883978 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.883952 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "189c7f88-3de8-4cce-872c-17ccf87b5885" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:38:01.883978 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.883959 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "189c7f88-3de8-4cce-872c-17ccf87b5885" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:38:01.884341 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.884319 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "189c7f88-3de8-4cce-872c-17ccf87b5885" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:38:01.884341 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.884327 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "189c7f88-3de8-4cce-872c-17ccf87b5885" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:38:01.889571 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.889544 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/189c7f88-3de8-4cce-872c-17ccf87b5885-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "189c7f88-3de8-4cce-872c-17ccf87b5885" (UID: "189c7f88-3de8-4cce-872c-17ccf87b5885"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:38:01.982687 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.982643 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m2jd\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-kube-api-access-5m2jd\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:01.982687 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.982678 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-image-registry-private-configuration\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:01.982687 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.982711 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:01.983008 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.982725 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-registry-certificates\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:01.983008 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.982738 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/189c7f88-3de8-4cce-872c-17ccf87b5885-installation-pull-secrets\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:01.983008 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.982750 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/189c7f88-3de8-4cce-872c-17ccf87b5885-trusted-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:01.983008 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.982761 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/189c7f88-3de8-4cce-872c-17ccf87b5885-ca-trust-extracted\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:01.983008 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:01.982772 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/189c7f88-3de8-4cce-872c-17ccf87b5885-bound-sa-token\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:02.183884 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:02.183849 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68f5ff5d87-l2ht2"] Jul 24 15:38:02.188546 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:02.188512 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68f5ff5d87-l2ht2"] Jul 24 15:38:03.249863 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:03.249831 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="189c7f88-3de8-4cce-872c-17ccf87b5885" path="/var/lib/kubelet/pods/189c7f88-3de8-4cce-872c-17ccf87b5885/volumes" Jul 24 15:38:07.887077 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:07.887044 2571 generic.go:356] "Generic (PLEG): container finished" podID="2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a" containerID="fd89369d3f1650988dacc6b59ca416782d76fcd46943bdfd4e15d7a3ff28c1db" exitCode=0 Jul 24 15:38:07.887503 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:07.887120 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" event={"ID":"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a","Type":"ContainerDied","Data":"fd89369d3f1650988dacc6b59ca416782d76fcd46943bdfd4e15d7a3ff28c1db"} Jul 24 15:38:07.887503 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:07.887422 2571 scope.go:117] "RemoveContainer" containerID="fd89369d3f1650988dacc6b59ca416782d76fcd46943bdfd4e15d7a3ff28c1db" Jul 24 15:38:08.892739 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:08.892688 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx" event={"ID":"2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a","Type":"ContainerStarted","Data":"f7f2f299f48a57c9bbd3cde28d0656eee5edbbf82c6b976ab72a2bbea8106586"} Jul 24 15:38:17.929577 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:17.929539 2571 generic.go:356] "Generic (PLEG): container finished" podID="9fbee300-3ce7-4e7f-8c17-e56643547910" containerID="15370fe0a70a8d684b0cfdca9426af2ac4944e0acf01c3075da61552021041ca" exitCode=0 Jul 24 15:38:17.930084 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:17.929622 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-br6wr" event={"ID":"9fbee300-3ce7-4e7f-8c17-e56643547910","Type":"ContainerDied","Data":"15370fe0a70a8d684b0cfdca9426af2ac4944e0acf01c3075da61552021041ca"} Jul 24 15:38:17.930084 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:17.930011 2571 scope.go:117] "RemoveContainer" containerID="15370fe0a70a8d684b0cfdca9426af2ac4944e0acf01c3075da61552021041ca" Jul 24 15:38:17.931187 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:17.931165 2571 generic.go:356] "Generic (PLEG): container finished" podID="6b977c05-00e9-4a94-8c33-7c57e8b953e1" containerID="6a0f6d33fef875004fc6fff9502f61d02714c9f555f09307ba8174f132464dc0" exitCode=0 Jul 24 15:38:17.931267 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:17.931194 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" event={"ID":"6b977c05-00e9-4a94-8c33-7c57e8b953e1","Type":"ContainerDied","Data":"6a0f6d33fef875004fc6fff9502f61d02714c9f555f09307ba8174f132464dc0"} Jul 24 15:38:17.931458 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:17.931445 2571 scope.go:117] "RemoveContainer" containerID="6a0f6d33fef875004fc6fff9502f61d02714c9f555f09307ba8174f132464dc0" Jul 24 15:38:18.936965 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:18.936924 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-br6wr" event={"ID":"9fbee300-3ce7-4e7f-8c17-e56643547910","Type":"ContainerStarted","Data":"2141a9442a5c34d990335ed9c52fcf6080e8c126d3a0b9fb2d72fd2bfe0a1a3e"} Jul 24 15:38:18.938809 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:18.938780 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-scscf" event={"ID":"6b977c05-00e9-4a94-8c33-7c57e8b953e1","Type":"ContainerStarted","Data":"b1bfe1d443247d3fb179f2abf0a7747cae89a6cf1e43b489db07ec2260ca6f97"} Jul 24 15:38:25.940839 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:25.940788 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d66b9dbc8-5l4sp" podUID="17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" containerName="console" containerID="cri-o://e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3" gracePeriod=15 Jul 24 15:38:26.222394 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.222326 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:38:26.277171 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277138 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-service-ca\") pod \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " Jul 24 15:38:26.277171 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277176 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-serving-cert\") pod \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " Jul 24 15:38:26.277439 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277225 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-oauth-serving-cert\") pod \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " Jul 24 15:38:26.277439 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277374 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-oauth-config\") pod \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " Jul 24 15:38:26.277542 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277459 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtbp4\" (UniqueName: \"kubernetes.io/projected/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-kube-api-access-wtbp4\") pod \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " Jul 24 15:38:26.277542 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277514 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-config\") pod \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\" (UID: \"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f\") " Jul 24 15:38:26.277663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277502 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-service-ca" (OuterVolumeSpecName: "service-ca") pod "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" (UID: "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:38:26.277744 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277554 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" (UID: "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:38:26.277906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277880 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-config" (OuterVolumeSpecName: "console-config") pod "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" (UID: "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:38:26.277980 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277935 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.277980 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.277949 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-oauth-serving-cert\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.279818 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.279794 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-kube-api-access-wtbp4" (OuterVolumeSpecName: "kube-api-access-wtbp4") pod "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" (UID: "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f"). InnerVolumeSpecName "kube-api-access-wtbp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:38:26.279904 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.279832 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" (UID: "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:38:26.280025 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.280006 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" (UID: "17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:38:26.379321 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.379281 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbp4\" (UniqueName: \"kubernetes.io/projected/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-kube-api-access-wtbp4\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.379321 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.379314 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-config\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.379321 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.379327 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-serving-cert\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.379554 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.379339 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f-console-oauth-config\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.972915 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.972882 2571 generic.go:356] "Generic (PLEG): container finished" podID="17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" containerID="e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3" exitCode=0 Jul 24 15:38:26.973339 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.972940 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d66b9dbc8-5l4sp" Jul 24 15:38:26.973339 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.972965 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d66b9dbc8-5l4sp" event={"ID":"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f","Type":"ContainerDied","Data":"e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3"} Jul 24 15:38:26.973339 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.973017 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d66b9dbc8-5l4sp" event={"ID":"17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f","Type":"ContainerDied","Data":"1b0a16467af4197ddeb4dc12086111f49f5379908f0e336f07b3b7638c7bc60c"} Jul 24 15:38:26.973339 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.973039 2571 scope.go:117] "RemoveContainer" containerID="e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3" Jul 24 15:38:26.981808 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.981796 2571 scope.go:117] "RemoveContainer" containerID="e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3" Jul 24 15:38:26.982081 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:38:26.982061 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3\": container with ID starting with e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3 not found: ID does not exist" containerID="e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3" Jul 24 15:38:26.982121 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.982090 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3"} err="failed to get container status \"e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3\": rpc error: code = NotFound desc = could not find container \"e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3\": container with ID starting with e92f6df9948b537bbe881cc269c31ff3051eaf48dd14a4fd82493f305537d2e3 not found: ID does not exist" Jul 24 15:38:26.995038 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.995012 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d66b9dbc8-5l4sp"] Jul 24 15:38:26.998925 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:26.998905 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d66b9dbc8-5l4sp"] Jul 24 15:38:27.249580 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:27.249551 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" path="/var/lib/kubelet/pods/17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f/volumes" Jul 24 15:38:50.577511 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.577362 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77f948c498-mxsf6"] Jul 24 15:38:50.578050 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.577828 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="189c7f88-3de8-4cce-872c-17ccf87b5885" containerName="registry" Jul 24 15:38:50.578050 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.577847 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="189c7f88-3de8-4cce-872c-17ccf87b5885" containerName="registry" Jul 24 15:38:50.578050 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.577877 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" containerName="console" Jul 24 15:38:50.578050 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.577885 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" containerName="console" Jul 24 15:38:50.578050 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.577953 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f7e2a6-cfaa-4f0a-935a-fc7b30fc466f" containerName="console" Jul 24 15:38:50.578050 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.577962 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="189c7f88-3de8-4cce-872c-17ccf87b5885" containerName="registry" Jul 24 15:38:50.579950 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.579924 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.594508 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.594482 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f948c498-mxsf6"] Jul 24 15:38:50.685996 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.685949 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-serving-cert\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.685996 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.685998 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-trusted-ca-bundle\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.686237 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.686044 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-oauth-config\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.686237 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.686087 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-oauth-serving-cert\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.686237 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.686139 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-service-ca\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.686237 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.686173 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bw4\" (UniqueName: \"kubernetes.io/projected/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-kube-api-access-n4bw4\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.686237 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.686215 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-config\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.787260 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.787219 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-oauth-serving-cert\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.787449 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.787283 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-service-ca\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.787449 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.787312 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bw4\" (UniqueName: \"kubernetes.io/projected/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-kube-api-access-n4bw4\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.787449 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.787350 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-config\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.787449 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.787381 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-serving-cert\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.787449 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.787416 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-trusted-ca-bundle\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.787449 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.787447 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-oauth-config\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.788159 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.788128 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-oauth-serving-cert\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.788352 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.788153 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-service-ca\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.788352 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.788161 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-config\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.788492 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.788285 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-trusted-ca-bundle\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.789996 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.789977 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-oauth-config\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.790091 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.790073 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-serving-cert\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.804337 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.804314 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bw4\" (UniqueName: \"kubernetes.io/projected/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-kube-api-access-n4bw4\") pod \"console-77f948c498-mxsf6\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:50.890237 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:50.890150 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:38:51.021420 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:51.021396 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f948c498-mxsf6"] Jul 24 15:38:51.023522 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:38:51.023492 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd70a16_04c1_4813_86cd_ae7a7e3b8d2b.slice/crio-f9aa6fef1cf8ccde442866c7e671d5112e95efd729903f6ef32f7af8c9a0967c WatchSource:0}: Error finding container f9aa6fef1cf8ccde442866c7e671d5112e95efd729903f6ef32f7af8c9a0967c: Status 404 returned error can't find the container with id f9aa6fef1cf8ccde442866c7e671d5112e95efd729903f6ef32f7af8c9a0967c Jul 24 15:38:51.065459 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:51.065426 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f948c498-mxsf6" event={"ID":"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b","Type":"ContainerStarted","Data":"f9aa6fef1cf8ccde442866c7e671d5112e95efd729903f6ef32f7af8c9a0967c"} Jul 24 15:38:52.070843 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:52.070804 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f948c498-mxsf6" event={"ID":"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b","Type":"ContainerStarted","Data":"478bf5449d0acbc911c49e73463713bca60816303f5b9a1bc83658acc998b2e2"} Jul 24 15:38:52.093115 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:38:52.093062 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77f948c498-mxsf6" podStartSLOduration=2.093045609 podStartE2EDuration="2.093045609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:38:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:38:52.092093101 +0000 UTC m=+173.324820778" watchObservedRunningTime="2026-07-24 15:38:52.093045609 +0000 UTC m=+173.325773286" Jul 24 15:39:00.891015 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:00.890968 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:39:00.891483 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:00.891030 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:39:00.895891 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:00.895868 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:39:01.109103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:01.109078 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:39:01.156113 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:01.156037 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdf49f974-fw5fg"] Jul 24 15:39:26.180350 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.180276 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5fdf49f974-fw5fg" podUID="e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" containerName="console" containerID="cri-o://03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa" gracePeriod=15 Jul 24 15:39:26.415206 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.415179 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:39:26.465245 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465162 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-service-ca\") pod \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " Jul 24 15:39:26.465245 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465202 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-oauth-serving-cert\") pod \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " Jul 24 15:39:26.465245 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465229 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-config\") pod \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " Jul 24 15:39:26.465245 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465244 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4dfs\" (UniqueName: \"kubernetes.io/projected/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-kube-api-access-h4dfs\") pod \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " Jul 24 15:39:26.465542 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465306 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-oauth-config\") pod \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " Jul 24 15:39:26.465542 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465334 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-serving-cert\") pod \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " Jul 24 15:39:26.465542 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465375 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-trusted-ca-bundle\") pod \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\" (UID: \"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1\") " Jul 24 15:39:26.465729 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465602 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-service-ca" (OuterVolumeSpecName: "service-ca") pod "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" (UID: "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:39:26.465729 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465619 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-config" (OuterVolumeSpecName: "console-config") pod "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" (UID: "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:39:26.465729 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465629 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" (UID: "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:39:26.465922 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.465899 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" (UID: "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:39:26.467727 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.467676 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" (UID: "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:39:26.468084 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.468060 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-kube-api-access-h4dfs" (OuterVolumeSpecName: "kube-api-access-h4dfs") pod "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" (UID: "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1"). InnerVolumeSpecName "kube-api-access-h4dfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:39:26.468184 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.468073 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" (UID: "e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:39:26.566374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.566325 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-trusted-ca-bundle\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:39:26.566374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.566368 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:39:26.566374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.566379 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-oauth-serving-cert\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:39:26.566374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.566387 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-config\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:39:26.566644 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.566396 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4dfs\" (UniqueName: \"kubernetes.io/projected/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-kube-api-access-h4dfs\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:39:26.566644 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.566405 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-oauth-config\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:39:26.566644 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:26.566414 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1-console-serving-cert\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:39:27.213557 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.213523 2571 generic.go:356] "Generic (PLEG): container finished" podID="e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" containerID="03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa" exitCode=0 Jul 24 15:39:27.213997 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.213590 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fdf49f974-fw5fg" Jul 24 15:39:27.213997 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.213616 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdf49f974-fw5fg" event={"ID":"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1","Type":"ContainerDied","Data":"03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa"} Jul 24 15:39:27.213997 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.213670 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fdf49f974-fw5fg" event={"ID":"e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1","Type":"ContainerDied","Data":"c902967aba133f7ec0a53f9b3dcf30bd20074250ceeb5bcbb99239bd61c85b6f"} Jul 24 15:39:27.213997 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.213711 2571 scope.go:117] "RemoveContainer" containerID="03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa" Jul 24 15:39:27.223203 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.223185 2571 scope.go:117] "RemoveContainer" containerID="03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa" Jul 24 15:39:27.223467 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:39:27.223450 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa\": container with ID starting with 03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa not found: ID does not exist" containerID="03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa" Jul 24 15:39:27.223571 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.223481 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa"} err="failed to get container status \"03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa\": rpc error: code = NotFound desc = could not find container \"03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa\": container with ID starting with 03b5cb4f3430e15b396bd44e19172a2ace0892631b97eb8d7e43f05e0aec7dfa not found: ID does not exist" Jul 24 15:39:27.236168 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.236141 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fdf49f974-fw5fg"] Jul 24 15:39:27.241437 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.241412 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fdf49f974-fw5fg"] Jul 24 15:39:27.249950 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:27.249927 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" path="/var/lib/kubelet/pods/e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1/volumes" Jul 24 15:39:53.690013 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.689975 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp"] Jul 24 15:39:53.690591 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.690380 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" containerName="console" Jul 24 15:39:53.690591 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.690396 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" containerName="console" Jul 24 15:39:53.690591 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.690479 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e16bddfc-50b5-4ad8-9ecf-dcb77fa7acd1" containerName="console" Jul 24 15:39:53.692595 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.692567 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:53.695465 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.695442 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 15:39:53.695579 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.695444 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ssdw4\"" Jul 24 15:39:53.696388 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.696361 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 15:39:53.709418 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.709389 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp"] Jul 24 15:39:53.776119 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.776085 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:53.776276 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.776179 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:53.776276 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.776206 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfl9z\" (UniqueName: \"kubernetes.io/projected/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-kube-api-access-jfl9z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:53.876970 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.876927 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:53.877167 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.877008 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:53.877167 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.877033 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfl9z\" (UniqueName: \"kubernetes.io/projected/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-kube-api-access-jfl9z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:53.877342 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.877320 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:53.877401 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.877343 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:53.887508 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:53.887473 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfl9z\" (UniqueName: \"kubernetes.io/projected/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-kube-api-access-jfl9z\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:54.002356 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:54.002323 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:39:54.136064 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:54.136038 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp"] Jul 24 15:39:54.138805 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:39:54.138779 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6080aa1c_1cf8_47f8_b2cb_c443496f7eaa.slice/crio-f25a6ad845dc104becf35505a0f50abb31a5ac1d64923df9153d4c507e2c9946 WatchSource:0}: Error finding container f25a6ad845dc104becf35505a0f50abb31a5ac1d64923df9153d4c507e2c9946: Status 404 returned error can't find the container with id f25a6ad845dc104becf35505a0f50abb31a5ac1d64923df9153d4c507e2c9946 Jul 24 15:39:54.313252 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:39:54.313163 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" event={"ID":"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa","Type":"ContainerStarted","Data":"f25a6ad845dc104becf35505a0f50abb31a5ac1d64923df9153d4c507e2c9946"} Jul 24 15:40:02.347375 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:02.347338 2571 generic.go:356] "Generic (PLEG): container finished" podID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerID="34468e36f353ce7c8b2a6435a50cb16d3588d2442294ef78f58f57f3d2b77c49" exitCode=0 Jul 24 15:40:02.347780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:02.347385 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" event={"ID":"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa","Type":"ContainerDied","Data":"34468e36f353ce7c8b2a6435a50cb16d3588d2442294ef78f58f57f3d2b77c49"} Jul 24 15:40:06.365032 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:06.364995 2571 generic.go:356] "Generic (PLEG): container finished" podID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerID="fed5dca904e34eccd0114f78b94741d5f2bea512602cdc1cad42e476418e7591" exitCode=0 Jul 24 15:40:06.365686 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:06.365080 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" event={"ID":"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa","Type":"ContainerDied","Data":"fed5dca904e34eccd0114f78b94741d5f2bea512602cdc1cad42e476418e7591"} Jul 24 15:40:14.406649 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:14.406613 2571 generic.go:356] "Generic (PLEG): container finished" podID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerID="01d9fe20b2e3883962765b95d3d54a0b9b1774dc81e8bbf56416faa07ed28234" exitCode=0 Jul 24 15:40:14.407100 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:14.406680 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" event={"ID":"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa","Type":"ContainerDied","Data":"01d9fe20b2e3883962765b95d3d54a0b9b1774dc81e8bbf56416faa07ed28234"} Jul 24 15:40:15.575407 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.575370 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:40:15.665742 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.665628 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-util\") pod \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " Jul 24 15:40:15.665742 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.665663 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-bundle\") pod \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " Jul 24 15:40:15.665742 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.665687 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfl9z\" (UniqueName: \"kubernetes.io/projected/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-kube-api-access-jfl9z\") pod \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\" (UID: \"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa\") " Jul 24 15:40:15.666407 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.666378 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-bundle" (OuterVolumeSpecName: "bundle") pod "6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" (UID: "6080aa1c-1cf8-47f8-b2cb-c443496f7eaa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:40:15.668111 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.668083 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-kube-api-access-jfl9z" (OuterVolumeSpecName: "kube-api-access-jfl9z") pod "6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" (UID: "6080aa1c-1cf8-47f8-b2cb-c443496f7eaa"). InnerVolumeSpecName "kube-api-access-jfl9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:40:15.669924 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.669906 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-util" (OuterVolumeSpecName: "util") pod "6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" (UID: "6080aa1c-1cf8-47f8-b2cb-c443496f7eaa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:40:15.766652 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.766619 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-util\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:40:15.766652 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.766646 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-bundle\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:40:15.766652 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:15.766658 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfl9z\" (UniqueName: \"kubernetes.io/projected/6080aa1c-1cf8-47f8-b2cb-c443496f7eaa-kube-api-access-jfl9z\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:40:16.415057 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:16.415018 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" event={"ID":"6080aa1c-1cf8-47f8-b2cb-c443496f7eaa","Type":"ContainerDied","Data":"f25a6ad845dc104becf35505a0f50abb31a5ac1d64923df9153d4c507e2c9946"} Jul 24 15:40:16.415224 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:16.415073 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f25a6ad845dc104becf35505a0f50abb31a5ac1d64923df9153d4c507e2c9946" Jul 24 15:40:16.415224 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:16.415034 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ngxsp" Jul 24 15:40:21.723492 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.723409 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z"] Jul 24 15:40:21.724046 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.723813 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerName="extract" Jul 24 15:40:21.724046 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.723829 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerName="extract" Jul 24 15:40:21.724046 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.723845 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerName="pull" Jul 24 15:40:21.724046 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.723852 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerName="pull" Jul 24 15:40:21.724046 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.723869 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerName="util" Jul 24 15:40:21.724046 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.723877 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerName="util" Jul 24 15:40:21.724046 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.723971 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6080aa1c-1cf8-47f8-b2cb-c443496f7eaa" containerName="extract" Jul 24 15:40:21.726153 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.726133 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" Jul 24 15:40:21.728984 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.728966 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-b8fg7\"" Jul 24 15:40:21.729069 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.729013 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 24 15:40:21.729115 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.729068 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:40:21.740548 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.740529 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z"] Jul 24 15:40:21.918763 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.918729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e3834b0-7f47-4dfc-ab65-47f866b2012c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-87g9z\" (UID: \"9e3834b0-7f47-4dfc-ab65-47f866b2012c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" Jul 24 15:40:21.918922 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:21.918785 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swklz\" (UniqueName: \"kubernetes.io/projected/9e3834b0-7f47-4dfc-ab65-47f866b2012c-kube-api-access-swklz\") pod \"cert-manager-operator-controller-manager-55c444bf86-87g9z\" (UID: \"9e3834b0-7f47-4dfc-ab65-47f866b2012c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" Jul 24 15:40:22.019533 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:22.019498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e3834b0-7f47-4dfc-ab65-47f866b2012c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-87g9z\" (UID: \"9e3834b0-7f47-4dfc-ab65-47f866b2012c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" Jul 24 15:40:22.019745 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:22.019568 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swklz\" (UniqueName: \"kubernetes.io/projected/9e3834b0-7f47-4dfc-ab65-47f866b2012c-kube-api-access-swklz\") pod \"cert-manager-operator-controller-manager-55c444bf86-87g9z\" (UID: \"9e3834b0-7f47-4dfc-ab65-47f866b2012c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" Jul 24 15:40:22.019892 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:22.019872 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e3834b0-7f47-4dfc-ab65-47f866b2012c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-87g9z\" (UID: \"9e3834b0-7f47-4dfc-ab65-47f866b2012c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" Jul 24 15:40:22.028899 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:22.028868 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swklz\" (UniqueName: \"kubernetes.io/projected/9e3834b0-7f47-4dfc-ab65-47f866b2012c-kube-api-access-swklz\") pod \"cert-manager-operator-controller-manager-55c444bf86-87g9z\" (UID: \"9e3834b0-7f47-4dfc-ab65-47f866b2012c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" Jul 24 15:40:22.035514 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:22.035497 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" Jul 24 15:40:22.206108 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:22.206067 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z"] Jul 24 15:40:22.214261 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:40:22.214231 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3834b0_7f47_4dfc_ab65_47f866b2012c.slice/crio-25ac7766e3825b3759fa5f335c34c2c8e23019f10f479c3251da87362c3de0d2 WatchSource:0}: Error finding container 25ac7766e3825b3759fa5f335c34c2c8e23019f10f479c3251da87362c3de0d2: Status 404 returned error can't find the container with id 25ac7766e3825b3759fa5f335c34c2c8e23019f10f479c3251da87362c3de0d2 Jul 24 15:40:22.440756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:22.440637 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" event={"ID":"9e3834b0-7f47-4dfc-ab65-47f866b2012c","Type":"ContainerStarted","Data":"25ac7766e3825b3759fa5f335c34c2c8e23019f10f479c3251da87362c3de0d2"} Jul 24 15:40:36.498959 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:36.498924 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" event={"ID":"9e3834b0-7f47-4dfc-ab65-47f866b2012c","Type":"ContainerStarted","Data":"b0a86a84440d71e64aecc282b55bbee2584421c77634ded53fc9e3a407bd9324"} Jul 24 15:40:36.522410 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:36.522358 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-87g9z" podStartSLOduration=1.5883643570000001 podStartE2EDuration="15.522342586s" totalImagesPullingTime="13.933978229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:40:36.520916826 +0000 UTC m=+277.753644504" watchObservedRunningTime="2026-07-24 15:40:36.522342586 +0000 UTC m=+277.755070263" Jul 24 15:40:37.989957 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:37.989921 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt"] Jul 24 15:40:37.992404 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:37.992382 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:37.995252 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:37.995226 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ssdw4\"" Jul 24 15:40:37.995377 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:37.995359 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 15:40:37.996393 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:37.996376 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 15:40:38.001917 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.001898 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt"] Jul 24 15:40:38.031059 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.031027 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.031180 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.031064 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbs2w\" (UniqueName: \"kubernetes.io/projected/9f725f31-238e-4516-ad5d-7d9327c79bde-kube-api-access-wbs2w\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.031180 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.031148 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.132259 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.132224 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.132428 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.132282 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.132428 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.132393 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbs2w\" (UniqueName: \"kubernetes.io/projected/9f725f31-238e-4516-ad5d-7d9327c79bde-kube-api-access-wbs2w\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.132608 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.132593 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.132675 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.132646 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.141540 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.141517 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbs2w\" (UniqueName: \"kubernetes.io/projected/9f725f31-238e-4516-ad5d-7d9327c79bde-kube-api-access-wbs2w\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.301383 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.301283 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:38.438346 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.435534 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt"] Jul 24 15:40:38.441346 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:40:38.441315 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f725f31_238e_4516_ad5d_7d9327c79bde.slice/crio-6a0021bfa5c78dd6ed6437bc2589e03aa8354c863239ba5444fb96697a8fafa4 WatchSource:0}: Error finding container 6a0021bfa5c78dd6ed6437bc2589e03aa8354c863239ba5444fb96697a8fafa4: Status 404 returned error can't find the container with id 6a0021bfa5c78dd6ed6437bc2589e03aa8354c863239ba5444fb96697a8fafa4 Jul 24 15:40:38.508382 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.508350 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" event={"ID":"9f725f31-238e-4516-ad5d-7d9327c79bde","Type":"ContainerStarted","Data":"494d087a7cb3b3d35ba7499262ed3ea5d73eeb9a6f73163cde508547205010e3"} Jul 24 15:40:38.508547 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:38.508402 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" event={"ID":"9f725f31-238e-4516-ad5d-7d9327c79bde","Type":"ContainerStarted","Data":"6a0021bfa5c78dd6ed6437bc2589e03aa8354c863239ba5444fb96697a8fafa4"} Jul 24 15:40:39.513760 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:39.513731 2571 generic.go:356] "Generic (PLEG): container finished" podID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerID="494d087a7cb3b3d35ba7499262ed3ea5d73eeb9a6f73163cde508547205010e3" exitCode=0 Jul 24 15:40:39.514152 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:39.513778 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" event={"ID":"9f725f31-238e-4516-ad5d-7d9327c79bde","Type":"ContainerDied","Data":"494d087a7cb3b3d35ba7499262ed3ea5d73eeb9a6f73163cde508547205010e3"} Jul 24 15:40:42.528590 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:42.528548 2571 generic.go:356] "Generic (PLEG): container finished" podID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerID="352d783545a77711910d9ed10ee45fff378712915c6f857cc3a770ee857b5df7" exitCode=0 Jul 24 15:40:42.529010 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:42.528637 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" event={"ID":"9f725f31-238e-4516-ad5d-7d9327c79bde","Type":"ContainerDied","Data":"352d783545a77711910d9ed10ee45fff378712915c6f857cc3a770ee857b5df7"} Jul 24 15:40:43.534358 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:43.534320 2571 generic.go:356] "Generic (PLEG): container finished" podID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerID="93563a1d4de3715917dd65d349593fa294b6cd326d1da2c6b708399663e7d4d7" exitCode=0 Jul 24 15:40:43.534779 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:43.534404 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" event={"ID":"9f725f31-238e-4516-ad5d-7d9327c79bde","Type":"ContainerDied","Data":"93563a1d4de3715917dd65d349593fa294b6cd326d1da2c6b708399663e7d4d7"} Jul 24 15:40:44.681735 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.681711 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:44.786663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.786628 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-util\") pod \"9f725f31-238e-4516-ad5d-7d9327c79bde\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " Jul 24 15:40:44.786859 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.786736 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbs2w\" (UniqueName: \"kubernetes.io/projected/9f725f31-238e-4516-ad5d-7d9327c79bde-kube-api-access-wbs2w\") pod \"9f725f31-238e-4516-ad5d-7d9327c79bde\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " Jul 24 15:40:44.786859 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.786785 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-bundle\") pod \"9f725f31-238e-4516-ad5d-7d9327c79bde\" (UID: \"9f725f31-238e-4516-ad5d-7d9327c79bde\") " Jul 24 15:40:44.787222 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.787193 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-bundle" (OuterVolumeSpecName: "bundle") pod "9f725f31-238e-4516-ad5d-7d9327c79bde" (UID: "9f725f31-238e-4516-ad5d-7d9327c79bde"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:40:44.788954 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.788898 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f725f31-238e-4516-ad5d-7d9327c79bde-kube-api-access-wbs2w" (OuterVolumeSpecName: "kube-api-access-wbs2w") pod "9f725f31-238e-4516-ad5d-7d9327c79bde" (UID: "9f725f31-238e-4516-ad5d-7d9327c79bde"). InnerVolumeSpecName "kube-api-access-wbs2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:40:44.791393 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.791357 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-util" (OuterVolumeSpecName: "util") pod "9f725f31-238e-4516-ad5d-7d9327c79bde" (UID: "9f725f31-238e-4516-ad5d-7d9327c79bde"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:40:44.887425 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.887398 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbs2w\" (UniqueName: \"kubernetes.io/projected/9f725f31-238e-4516-ad5d-7d9327c79bde-kube-api-access-wbs2w\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:40:44.887425 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.887421 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-bundle\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:40:44.887425 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:44.887430 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f725f31-238e-4516-ad5d-7d9327c79bde-util\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:40:45.179262 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.179181 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs"] Jul 24 15:40:45.179518 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.179505 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerName="extract" Jul 24 15:40:45.179565 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.179520 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerName="extract" Jul 24 15:40:45.179565 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.179534 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerName="util" Jul 24 15:40:45.179565 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.179540 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerName="util" Jul 24 15:40:45.179565 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.179556 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerName="pull" Jul 24 15:40:45.179565 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.179561 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerName="pull" Jul 24 15:40:45.179804 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.179627 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f725f31-238e-4516-ad5d-7d9327c79bde" containerName="extract" Jul 24 15:40:45.195141 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.195114 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs"] Jul 24 15:40:45.195272 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.195235 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" Jul 24 15:40:45.197805 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.197783 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 15:40:45.197923 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.197785 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-5gzb2\"" Jul 24 15:40:45.198817 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.198798 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 15:40:45.291322 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.291279 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4zd\" (UniqueName: \"kubernetes.io/projected/d2509316-224b-49dc-a536-93871e78ef05-kube-api-access-4d4zd\") pod \"cert-manager-cainjector-559cdd8c87-5dmhs\" (UID: \"d2509316-224b-49dc-a536-93871e78ef05\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" Jul 24 15:40:45.291477 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.291344 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2509316-224b-49dc-a536-93871e78ef05-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5dmhs\" (UID: \"d2509316-224b-49dc-a536-93871e78ef05\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" Jul 24 15:40:45.392458 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.392420 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2509316-224b-49dc-a536-93871e78ef05-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5dmhs\" (UID: \"d2509316-224b-49dc-a536-93871e78ef05\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" Jul 24 15:40:45.392635 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.392483 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4zd\" (UniqueName: \"kubernetes.io/projected/d2509316-224b-49dc-a536-93871e78ef05-kube-api-access-4d4zd\") pod \"cert-manager-cainjector-559cdd8c87-5dmhs\" (UID: \"d2509316-224b-49dc-a536-93871e78ef05\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" Jul 24 15:40:45.400984 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.400960 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d2509316-224b-49dc-a536-93871e78ef05-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-5dmhs\" (UID: \"d2509316-224b-49dc-a536-93871e78ef05\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" Jul 24 15:40:45.401118 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.401100 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4zd\" (UniqueName: \"kubernetes.io/projected/d2509316-224b-49dc-a536-93871e78ef05-kube-api-access-4d4zd\") pod \"cert-manager-cainjector-559cdd8c87-5dmhs\" (UID: \"d2509316-224b-49dc-a536-93871e78ef05\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" Jul 24 15:40:45.505222 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.505189 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" Jul 24 15:40:45.544808 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.544773 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" event={"ID":"9f725f31-238e-4516-ad5d-7d9327c79bde","Type":"ContainerDied","Data":"6a0021bfa5c78dd6ed6437bc2589e03aa8354c863239ba5444fb96697a8fafa4"} Jul 24 15:40:45.544808 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.544804 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5dznt" Jul 24 15:40:45.544986 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.544820 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a0021bfa5c78dd6ed6437bc2589e03aa8354c863239ba5444fb96697a8fafa4" Jul 24 15:40:45.656933 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:45.656868 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs"] Jul 24 15:40:45.659460 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:40:45.659432 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2509316_224b_49dc_a536_93871e78ef05.slice/crio-49585affd6bdd50a01e64011cc0cb2b599ef438461ba41aa4baf11b6891cfa34 WatchSource:0}: Error finding container 49585affd6bdd50a01e64011cc0cb2b599ef438461ba41aa4baf11b6891cfa34: Status 404 returned error can't find the container with id 49585affd6bdd50a01e64011cc0cb2b599ef438461ba41aa4baf11b6891cfa34 Jul 24 15:40:46.550861 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:46.550815 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" event={"ID":"d2509316-224b-49dc-a536-93871e78ef05","Type":"ContainerStarted","Data":"49585affd6bdd50a01e64011cc0cb2b599ef438461ba41aa4baf11b6891cfa34"} Jul 24 15:40:48.560893 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:48.560860 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" event={"ID":"d2509316-224b-49dc-a536-93871e78ef05","Type":"ContainerStarted","Data":"fb79826494127e9aa725608ec4715edce8830b518d1549432b26c4bdf6cb9f6b"} Jul 24 15:40:48.579917 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:48.579868 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-5dmhs" podStartSLOduration=0.762499594 podStartE2EDuration="3.579851035s" totalImagesPullingTime="2.817351441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:40:48.578689095 +0000 UTC m=+289.811416774" watchObservedRunningTime="2026-07-24 15:40:48.579851035 +0000 UTC m=+289.812578712" Jul 24 15:40:57.668391 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.668341 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf"] Jul 24 15:40:57.670646 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.670631 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.673199 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.673179 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 24 15:40:57.673199 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.673190 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 24 15:40:57.674278 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.674262 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ssdw4\"" Jul 24 15:40:57.679802 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.679779 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf"] Jul 24 15:40:57.794489 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.794460 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.794670 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.794505 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.794670 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.794568 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbmdw\" (UniqueName: \"kubernetes.io/projected/bc5ee845-3786-4c39-890f-20e8aa9808fd-kube-api-access-vbmdw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.895759 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.895727 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.895912 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.895773 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.895912 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.895811 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbmdw\" (UniqueName: \"kubernetes.io/projected/bc5ee845-3786-4c39-890f-20e8aa9808fd-kube-api-access-vbmdw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.896122 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.896103 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.896170 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.896152 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.905373 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.905337 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbmdw\" (UniqueName: \"kubernetes.io/projected/bc5ee845-3786-4c39-890f-20e8aa9808fd-kube-api-access-vbmdw\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:57.980672 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:57.980597 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:40:58.110329 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:58.110299 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf"] Jul 24 15:40:58.112678 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:40:58.112639 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5ee845_3786_4c39_890f_20e8aa9808fd.slice/crio-a3662484ba6a1723069786e0d3a6f64454da3dc50f4892f95aa3f31a098dd3d5 WatchSource:0}: Error finding container a3662484ba6a1723069786e0d3a6f64454da3dc50f4892f95aa3f31a098dd3d5: Status 404 returned error can't find the container with id a3662484ba6a1723069786e0d3a6f64454da3dc50f4892f95aa3f31a098dd3d5 Jul 24 15:40:58.604408 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:58.604370 2571 generic.go:356] "Generic (PLEG): container finished" podID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerID="b53c7b82efe62ebf96cc6f666b6d4113b7443d48c00c07870df4f7cb5551efc7" exitCode=0 Jul 24 15:40:58.604588 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:58.604453 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" event={"ID":"bc5ee845-3786-4c39-890f-20e8aa9808fd","Type":"ContainerDied","Data":"b53c7b82efe62ebf96cc6f666b6d4113b7443d48c00c07870df4f7cb5551efc7"} Jul 24 15:40:58.604588 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:58.604504 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" event={"ID":"bc5ee845-3786-4c39-890f-20e8aa9808fd","Type":"ContainerStarted","Data":"a3662484ba6a1723069786e0d3a6f64454da3dc50f4892f95aa3f31a098dd3d5"} Jul 24 15:40:59.132643 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:59.132618 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:40:59.133430 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:59.133407 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:40:59.138960 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:59.138940 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:40:59.139568 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:59.139553 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:40:59.145141 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:59.145123 2571 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 15:40:59.611011 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:59.610981 2571 generic.go:356] "Generic (PLEG): container finished" podID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerID="79e35ed3e7c4d10b53cb59d65d437210dbc69772e672554a8e3e6e75b29b0380" exitCode=0 Jul 24 15:40:59.612110 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:40:59.611056 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" event={"ID":"bc5ee845-3786-4c39-890f-20e8aa9808fd","Type":"ContainerDied","Data":"79e35ed3e7c4d10b53cb59d65d437210dbc69772e672554a8e3e6e75b29b0380"} Jul 24 15:41:00.617843 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:00.617810 2571 generic.go:356] "Generic (PLEG): container finished" podID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerID="565f38b7e65a89855c61586acb0e2df3ccc84193d2c2de0bac0bd13ad3b9f019" exitCode=0 Jul 24 15:41:00.618297 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:00.617905 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" event={"ID":"bc5ee845-3786-4c39-890f-20e8aa9808fd","Type":"ContainerDied","Data":"565f38b7e65a89855c61586acb0e2df3ccc84193d2c2de0bac0bd13ad3b9f019"} Jul 24 15:41:01.749069 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.749042 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:41:01.830647 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.830610 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-bundle\") pod \"bc5ee845-3786-4c39-890f-20e8aa9808fd\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " Jul 24 15:41:01.830866 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.830676 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-util\") pod \"bc5ee845-3786-4c39-890f-20e8aa9808fd\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " Jul 24 15:41:01.830866 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.830731 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbmdw\" (UniqueName: \"kubernetes.io/projected/bc5ee845-3786-4c39-890f-20e8aa9808fd-kube-api-access-vbmdw\") pod \"bc5ee845-3786-4c39-890f-20e8aa9808fd\" (UID: \"bc5ee845-3786-4c39-890f-20e8aa9808fd\") " Jul 24 15:41:01.831463 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.831433 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-bundle" (OuterVolumeSpecName: "bundle") pod "bc5ee845-3786-4c39-890f-20e8aa9808fd" (UID: "bc5ee845-3786-4c39-890f-20e8aa9808fd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:41:01.832995 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.832966 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5ee845-3786-4c39-890f-20e8aa9808fd-kube-api-access-vbmdw" (OuterVolumeSpecName: "kube-api-access-vbmdw") pod "bc5ee845-3786-4c39-890f-20e8aa9808fd" (UID: "bc5ee845-3786-4c39-890f-20e8aa9808fd"). InnerVolumeSpecName "kube-api-access-vbmdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:41:01.836256 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.836238 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-util" (OuterVolumeSpecName: "util") pod "bc5ee845-3786-4c39-890f-20e8aa9808fd" (UID: "bc5ee845-3786-4c39-890f-20e8aa9808fd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:41:01.932084 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.932000 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-bundle\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:41:01.932084 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.932028 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc5ee845-3786-4c39-890f-20e8aa9808fd-util\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:41:01.932084 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:01.932036 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbmdw\" (UniqueName: \"kubernetes.io/projected/bc5ee845-3786-4c39-890f-20e8aa9808fd-kube-api-access-vbmdw\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:41:02.632313 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:02.632275 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" event={"ID":"bc5ee845-3786-4c39-890f-20e8aa9808fd","Type":"ContainerDied","Data":"a3662484ba6a1723069786e0d3a6f64454da3dc50f4892f95aa3f31a098dd3d5"} Jul 24 15:41:02.632531 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:02.632330 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3662484ba6a1723069786e0d3a6f64454da3dc50f4892f95aa3f31a098dd3d5" Jul 24 15:41:02.632531 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:02.632293 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afnlgxf" Jul 24 15:41:10.805419 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.805381 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc"] Jul 24 15:41:10.805918 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.805819 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerName="util" Jul 24 15:41:10.805918 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.805837 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerName="util" Jul 24 15:41:10.805918 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.805848 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerName="extract" Jul 24 15:41:10.805918 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.805856 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerName="extract" Jul 24 15:41:10.805918 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.805877 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerName="pull" Jul 24 15:41:10.805918 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.805885 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerName="pull" Jul 24 15:41:10.806212 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.805962 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc5ee845-3786-4c39-890f-20e8aa9808fd" containerName="extract" Jul 24 15:41:10.811709 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.811672 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:10.815256 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.815234 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 15:41:10.815467 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.815443 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-bk4ws\"" Jul 24 15:41:10.815528 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.815463 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 15:41:10.815586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.815570 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 15:41:10.815633 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.815596 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 15:41:10.822393 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.822367 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc"] Jul 24 15:41:10.905820 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.905785 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/461fa2c5-382e-4a09-99b5-0198fb86d1f8-tmp\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:10.905820 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.905823 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/461fa2c5-382e-4a09-99b5-0198fb86d1f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:10.906061 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.905855 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbmn\" (UniqueName: \"kubernetes.io/projected/461fa2c5-382e-4a09-99b5-0198fb86d1f8-kube-api-access-tqbmn\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:10.906061 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.905924 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/461fa2c5-382e-4a09-99b5-0198fb86d1f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:10.906061 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:10.905979 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/461fa2c5-382e-4a09-99b5-0198fb86d1f8-manifests\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.006916 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.006874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/461fa2c5-382e-4a09-99b5-0198fb86d1f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.007108 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.006935 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbmn\" (UniqueName: \"kubernetes.io/projected/461fa2c5-382e-4a09-99b5-0198fb86d1f8-kube-api-access-tqbmn\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.007108 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.006962 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/461fa2c5-382e-4a09-99b5-0198fb86d1f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.007108 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.006996 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/461fa2c5-382e-4a09-99b5-0198fb86d1f8-manifests\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.007108 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.007057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/461fa2c5-382e-4a09-99b5-0198fb86d1f8-tmp\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.007416 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.007395 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/461fa2c5-382e-4a09-99b5-0198fb86d1f8-manifests\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.009359 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.009336 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/461fa2c5-382e-4a09-99b5-0198fb86d1f8-tmp\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.009551 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.009531 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/461fa2c5-382e-4a09-99b5-0198fb86d1f8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.009607 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.009583 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/461fa2c5-382e-4a09-99b5-0198fb86d1f8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.016686 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.016659 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbmn\" (UniqueName: \"kubernetes.io/projected/461fa2c5-382e-4a09-99b5-0198fb86d1f8-kube-api-access-tqbmn\") pod \"opendatahub-operator-controller-manager-7b7484f4dc-fx5sc\" (UID: \"461fa2c5-382e-4a09-99b5-0198fb86d1f8\") " pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.123023 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.122937 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:11.263472 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.263449 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc"] Jul 24 15:41:11.265626 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:41:11.265600 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461fa2c5_382e_4a09_99b5_0198fb86d1f8.slice/crio-4515104f378a93153bcd5a1ae5863e22c30486f325f1d387c608522b20114365 WatchSource:0}: Error finding container 4515104f378a93153bcd5a1ae5863e22c30486f325f1d387c608522b20114365: Status 404 returned error can't find the container with id 4515104f378a93153bcd5a1ae5863e22c30486f325f1d387c608522b20114365 Jul 24 15:41:11.267277 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.267259 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:41:11.670640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:11.670601 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" event={"ID":"461fa2c5-382e-4a09-99b5-0198fb86d1f8","Type":"ContainerStarted","Data":"4515104f378a93153bcd5a1ae5863e22c30486f325f1d387c608522b20114365"} Jul 24 15:41:14.687058 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:14.686968 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" event={"ID":"461fa2c5-382e-4a09-99b5-0198fb86d1f8","Type":"ContainerStarted","Data":"154c919a4c07015fb8c001a5c0da32a514eded70468dacc4363cc4ca37ac2289"} Jul 24 15:41:15.693128 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:15.693029 2571 generic.go:356] "Generic (PLEG): container finished" podID="461fa2c5-382e-4a09-99b5-0198fb86d1f8" containerID="154c919a4c07015fb8c001a5c0da32a514eded70468dacc4363cc4ca37ac2289" exitCode=0 Jul 24 15:41:15.693128 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:15.693077 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" event={"ID":"461fa2c5-382e-4a09-99b5-0198fb86d1f8","Type":"ContainerDied","Data":"154c919a4c07015fb8c001a5c0da32a514eded70468dacc4363cc4ca37ac2289"} Jul 24 15:41:16.699743 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:16.699685 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" event={"ID":"461fa2c5-382e-4a09-99b5-0198fb86d1f8","Type":"ContainerStarted","Data":"b55c11493aa433762577f8094f5a880061d24ce44a5987a00799d3b7993b7dd7"} Jul 24 15:41:16.700135 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:16.699866 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:16.723850 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:16.723799 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" podStartSLOduration=2.259972043 podStartE2EDuration="6.72378417s" totalImagesPullingTime="3.375198401s" totalInitContainerRuntime="1.088613726s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:16.721317318 +0000 UTC m=+317.954044995" watchObservedRunningTime="2026-07-24 15:41:16.72378417 +0000 UTC m=+317.956511847" Jul 24 15:41:27.706409 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:27.706380 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7b7484f4dc-fx5sc" Jul 24 15:41:30.857799 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.857757 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-65d6g"] Jul 24 15:41:30.862499 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.862479 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf"] Jul 24 15:41:30.862623 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.862605 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:30.864304 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.864283 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:30.873165 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.873135 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nxzj\" (UniqueName: \"kubernetes.io/projected/a8572c57-9ea8-4200-9418-e0f93d913329-kube-api-access-8nxzj\") pod \"kube-auth-proxy-59fb5479dc-q7nxf\" (UID: \"a8572c57-9ea8-4200-9418-e0f93d913329\") " pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:30.873299 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.873260 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8572c57-9ea8-4200-9418-e0f93d913329-tmp\") pod \"kube-auth-proxy-59fb5479dc-q7nxf\" (UID: \"a8572c57-9ea8-4200-9418-e0f93d913329\") " pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:30.873366 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.873309 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8572c57-9ea8-4200-9418-e0f93d913329-tls-certs\") pod \"kube-auth-proxy-59fb5479dc-q7nxf\" (UID: \"a8572c57-9ea8-4200-9418-e0f93d913329\") " pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:30.876663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.876638 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-b7nwk\"" Jul 24 15:41:30.877647 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.877631 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 24 15:41:30.881993 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.881976 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 15:41:30.882320 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.882304 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 15:41:30.889322 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.889303 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-65d6g"] Jul 24 15:41:30.912891 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.912850 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf"] Jul 24 15:41:30.974434 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.974395 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8572c57-9ea8-4200-9418-e0f93d913329-tls-certs\") pod \"kube-auth-proxy-59fb5479dc-q7nxf\" (UID: \"a8572c57-9ea8-4200-9418-e0f93d913329\") " pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:30.974434 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.974436 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nxzj\" (UniqueName: \"kubernetes.io/projected/a8572c57-9ea8-4200-9418-e0f93d913329-kube-api-access-8nxzj\") pod \"kube-auth-proxy-59fb5479dc-q7nxf\" (UID: \"a8572c57-9ea8-4200-9418-e0f93d913329\") " pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:30.974663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.974469 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn2h8\" (UniqueName: \"kubernetes.io/projected/7dad3b4b-7b52-4554-9897-513cbd4b394d-kube-api-access-vn2h8\") pod \"odh-model-controller-7998c67dc4-65d6g\" (UID: \"7dad3b4b-7b52-4554-9897-513cbd4b394d\") " pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:30.974663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.974507 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8572c57-9ea8-4200-9418-e0f93d913329-tmp\") pod \"kube-auth-proxy-59fb5479dc-q7nxf\" (UID: \"a8572c57-9ea8-4200-9418-e0f93d913329\") " pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:30.974663 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.974522 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dad3b4b-7b52-4554-9897-513cbd4b394d-cert\") pod \"odh-model-controller-7998c67dc4-65d6g\" (UID: \"7dad3b4b-7b52-4554-9897-513cbd4b394d\") " pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:30.976963 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.976936 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8572c57-9ea8-4200-9418-e0f93d913329-tmp\") pod \"kube-auth-proxy-59fb5479dc-q7nxf\" (UID: \"a8572c57-9ea8-4200-9418-e0f93d913329\") " pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:30.977138 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.977122 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8572c57-9ea8-4200-9418-e0f93d913329-tls-certs\") pod \"kube-auth-proxy-59fb5479dc-q7nxf\" (UID: \"a8572c57-9ea8-4200-9418-e0f93d913329\") " pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:30.989903 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:30.989870 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nxzj\" (UniqueName: \"kubernetes.io/projected/a8572c57-9ea8-4200-9418-e0f93d913329-kube-api-access-8nxzj\") pod \"kube-auth-proxy-59fb5479dc-q7nxf\" (UID: \"a8572c57-9ea8-4200-9418-e0f93d913329\") " pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:31.075483 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.075448 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr"] Jul 24 15:41:31.075823 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.075800 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn2h8\" (UniqueName: \"kubernetes.io/projected/7dad3b4b-7b52-4554-9897-513cbd4b394d-kube-api-access-vn2h8\") pod \"odh-model-controller-7998c67dc4-65d6g\" (UID: \"7dad3b4b-7b52-4554-9897-513cbd4b394d\") " pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:31.075928 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.075882 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dad3b4b-7b52-4554-9897-513cbd4b394d-cert\") pod \"odh-model-controller-7998c67dc4-65d6g\" (UID: \"7dad3b4b-7b52-4554-9897-513cbd4b394d\") " pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:31.076024 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:41:31.076009 2571 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 24 15:41:31.076094 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:41:31.076085 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dad3b4b-7b52-4554-9897-513cbd4b394d-cert podName:7dad3b4b-7b52-4554-9897-513cbd4b394d nodeName:}" failed. No retries permitted until 2026-07-24 15:41:31.576066096 +0000 UTC m=+332.808793770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7dad3b4b-7b52-4554-9897-513cbd4b394d-cert") pod "odh-model-controller-7998c67dc4-65d6g" (UID: "7dad3b4b-7b52-4554-9897-513cbd4b394d") : secret "odh-model-controller-webhook-cert" not found Jul 24 15:41:31.077813 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.077797 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.081533 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.081512 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-txgss\"" Jul 24 15:41:31.081806 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.081788 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 15:41:31.081891 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.081788 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 15:41:31.081891 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.081813 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 15:41:31.081891 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.081809 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:41:31.081891 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.081832 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 15:41:31.088620 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.088598 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr"] Jul 24 15:41:31.090646 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.090619 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn2h8\" (UniqueName: \"kubernetes.io/projected/7dad3b4b-7b52-4554-9897-513cbd4b394d-kube-api-access-vn2h8\") pod \"odh-model-controller-7998c67dc4-65d6g\" (UID: \"7dad3b4b-7b52-4554-9897-513cbd4b394d\") " pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:31.176328 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.176241 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-metrics-cert\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.176328 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.176289 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6lmf\" (UniqueName: \"kubernetes.io/projected/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-kube-api-access-t6lmf\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.176513 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.176399 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-manager-config\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.176513 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.176420 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-cert\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.179053 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.179023 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" Jul 24 15:41:31.277258 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.277225 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-metrics-cert\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.277406 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.277301 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6lmf\" (UniqueName: \"kubernetes.io/projected/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-kube-api-access-t6lmf\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.277406 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.277394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-manager-config\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.277497 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.277418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-cert\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.278212 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.278183 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-manager-config\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.279993 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.279971 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-metrics-cert\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.280108 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.280020 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-cert\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.291024 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.290991 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6lmf\" (UniqueName: \"kubernetes.io/projected/3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8-kube-api-access-t6lmf\") pod \"lws-controller-manager-7db5d47bff-b9qvr\" (UID: \"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8\") " pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.323154 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.323126 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf"] Jul 24 15:41:31.325754 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:41:31.325725 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8572c57_9ea8_4200_9418_e0f93d913329.slice/crio-36e4839aaec9a6ce4d1f80f9c340b2fdb2bf4e3ec6e8f76c4509a7c6f0dcfe02 WatchSource:0}: Error finding container 36e4839aaec9a6ce4d1f80f9c340b2fdb2bf4e3ec6e8f76c4509a7c6f0dcfe02: Status 404 returned error can't find the container with id 36e4839aaec9a6ce4d1f80f9c340b2fdb2bf4e3ec6e8f76c4509a7c6f0dcfe02 Jul 24 15:41:31.388030 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.387999 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:31.530150 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.530124 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr"] Jul 24 15:41:31.532558 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:41:31.532509 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8bd6c4_1c68_45c3_b13e_9abb6ba0d4d8.slice/crio-81585a12b7cae978044c6b3c8e24492218b36577c3c7f88890c5ba02f2241dcd WatchSource:0}: Error finding container 81585a12b7cae978044c6b3c8e24492218b36577c3c7f88890c5ba02f2241dcd: Status 404 returned error can't find the container with id 81585a12b7cae978044c6b3c8e24492218b36577c3c7f88890c5ba02f2241dcd Jul 24 15:41:31.579570 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.579538 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dad3b4b-7b52-4554-9897-513cbd4b394d-cert\") pod \"odh-model-controller-7998c67dc4-65d6g\" (UID: \"7dad3b4b-7b52-4554-9897-513cbd4b394d\") " pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:31.582296 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.582216 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7dad3b4b-7b52-4554-9897-513cbd4b394d-cert\") pod \"odh-model-controller-7998c67dc4-65d6g\" (UID: \"7dad3b4b-7b52-4554-9897-513cbd4b394d\") " pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:31.771593 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.771552 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" event={"ID":"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8","Type":"ContainerStarted","Data":"81585a12b7cae978044c6b3c8e24492218b36577c3c7f88890c5ba02f2241dcd"} Jul 24 15:41:31.772781 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.772742 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" event={"ID":"a8572c57-9ea8-4200-9418-e0f93d913329","Type":"ContainerStarted","Data":"36e4839aaec9a6ce4d1f80f9c340b2fdb2bf4e3ec6e8f76c4509a7c6f0dcfe02"} Jul 24 15:41:31.773944 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.773926 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:31.945275 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:31.945238 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-65d6g"] Jul 24 15:41:31.960360 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:41:31.960329 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dad3b4b_7b52_4554_9897_513cbd4b394d.slice/crio-99e6fd97c1c804c70f0b4649e088db5c57c580505f85b5e099146500aadaf794 WatchSource:0}: Error finding container 99e6fd97c1c804c70f0b4649e088db5c57c580505f85b5e099146500aadaf794: Status 404 returned error can't find the container with id 99e6fd97c1c804c70f0b4649e088db5c57c580505f85b5e099146500aadaf794 Jul 24 15:41:32.791123 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:32.791085 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" event={"ID":"7dad3b4b-7b52-4554-9897-513cbd4b394d","Type":"ContainerStarted","Data":"99e6fd97c1c804c70f0b4649e088db5c57c580505f85b5e099146500aadaf794"} Jul 24 15:41:34.802878 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:34.802846 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" event={"ID":"3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8","Type":"ContainerStarted","Data":"3c32dd63cc46f03aae46f504274f5230bedd4b46a5b87a45a3ebcb093d4cbf1a"} Jul 24 15:41:34.803281 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:34.802960 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:34.819792 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:34.819744 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" podStartSLOduration=1.93150253 podStartE2EDuration="3.81972828s" totalImagesPullingTime="1.88822575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:34.818450186 +0000 UTC m=+336.051177866" watchObservedRunningTime="2026-07-24 15:41:34.81972828 +0000 UTC m=+336.052455957" Jul 24 15:41:35.809049 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:35.808995 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" event={"ID":"7dad3b4b-7b52-4554-9897-513cbd4b394d","Type":"ContainerStarted","Data":"b5dbc029def3bac0a34ae2db9d06fd23e94928b417a24ccd04c1ce778da1df7d"} Jul 24 15:41:35.809549 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:35.809153 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:41:35.810624 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:35.810599 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" event={"ID":"a8572c57-9ea8-4200-9418-e0f93d913329","Type":"ContainerStarted","Data":"19b156fb4127b86ed01ce078a59ff513522062c3151545c112510561e4f451ac"} Jul 24 15:41:35.825808 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:35.825758 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" podStartSLOduration=2.364518558 podStartE2EDuration="5.825742899s" totalImagesPullingTime="3.461224341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:35.824401614 +0000 UTC m=+337.057129329" watchObservedRunningTime="2026-07-24 15:41:35.825742899 +0000 UTC m=+337.058470574" Jul 24 15:41:35.842266 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:35.842207 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59fb5479dc-q7nxf" podStartSLOduration=1.791190179 podStartE2EDuration="5.842190542s" totalImagesPullingTime="4.051000363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:35.839448432 +0000 UTC m=+337.072176111" watchObservedRunningTime="2026-07-24 15:41:35.842190542 +0000 UTC m=+337.074918219" Jul 24 15:41:45.813514 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:45.813470 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7db5d47bff-b9qvr" Jul 24 15:41:46.818482 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:41:46.818451 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-65d6g" Jul 24 15:42:41.028967 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.028928 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85469664c6-pft48"] Jul 24 15:42:41.031865 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.031843 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.043656 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.043599 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85469664c6-pft48"] Jul 24 15:42:41.052447 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.052420 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-trusted-ca-bundle\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.052591 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.052456 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-console-config\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.052591 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.052475 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h888j\" (UniqueName: \"kubernetes.io/projected/2b695f2e-26d0-4599-aba0-b39228388794-kube-api-access-h888j\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.052591 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.052557 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-service-ca\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.052740 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.052629 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b695f2e-26d0-4599-aba0-b39228388794-console-serving-cert\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.052740 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.052676 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-oauth-serving-cert\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.052740 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.052729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b695f2e-26d0-4599-aba0-b39228388794-console-oauth-config\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.153131 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.153093 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-trusted-ca-bundle\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.153131 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.153135 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-console-config\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.153374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.153152 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h888j\" (UniqueName: \"kubernetes.io/projected/2b695f2e-26d0-4599-aba0-b39228388794-kube-api-access-h888j\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.153374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.153186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-service-ca\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.153374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.153210 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b695f2e-26d0-4599-aba0-b39228388794-console-serving-cert\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.153374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.153236 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-oauth-serving-cert\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.153374 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.153257 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b695f2e-26d0-4599-aba0-b39228388794-console-oauth-config\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.154007 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.153982 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-console-config\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.154184 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.154122 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-trusted-ca-bundle\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.154184 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.154121 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-service-ca\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.154322 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.154238 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b695f2e-26d0-4599-aba0-b39228388794-oauth-serving-cert\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.155811 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.155783 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b695f2e-26d0-4599-aba0-b39228388794-console-serving-cert\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.155886 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.155851 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b695f2e-26d0-4599-aba0-b39228388794-console-oauth-config\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.161858 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.161834 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h888j\" (UniqueName: \"kubernetes.io/projected/2b695f2e-26d0-4599-aba0-b39228388794-kube-api-access-h888j\") pod \"console-85469664c6-pft48\" (UID: \"2b695f2e-26d0-4599-aba0-b39228388794\") " pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.343452 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.343354 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:41.492167 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:41.492143 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85469664c6-pft48"] Jul 24 15:42:41.494095 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:42:41.494067 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b695f2e_26d0_4599_aba0_b39228388794.slice/crio-793ebc3c7103fc4b332cb7bba9523b2a258f331f014bd3a873f8e641537fb0be WatchSource:0}: Error finding container 793ebc3c7103fc4b332cb7bba9523b2a258f331f014bd3a873f8e641537fb0be: Status 404 returned error can't find the container with id 793ebc3c7103fc4b332cb7bba9523b2a258f331f014bd3a873f8e641537fb0be Jul 24 15:42:42.114123 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:42.114086 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85469664c6-pft48" event={"ID":"2b695f2e-26d0-4599-aba0-b39228388794","Type":"ContainerStarted","Data":"4e1be6b6e35232bc907f8b5dbe272e4b55455c015efaac66cd0df1e286e04bb5"} Jul 24 15:42:42.114516 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:42.114142 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85469664c6-pft48" event={"ID":"2b695f2e-26d0-4599-aba0-b39228388794","Type":"ContainerStarted","Data":"793ebc3c7103fc4b332cb7bba9523b2a258f331f014bd3a873f8e641537fb0be"} Jul 24 15:42:42.131521 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:42.131465 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85469664c6-pft48" podStartSLOduration=1.131449365 podStartE2EDuration="1.131449365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:42.131287788 +0000 UTC m=+403.364015489" watchObservedRunningTime="2026-07-24 15:42:42.131449365 +0000 UTC m=+403.364177042" Jul 24 15:42:47.815209 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:47.815164 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb"] Jul 24 15:42:47.818103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:47.818081 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:47.821046 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:47.821014 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:42:47.821166 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:47.821070 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:42:47.821166 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:47.821016 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9ds7q\"" Jul 24 15:42:47.829175 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:47.829156 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb"] Jul 24 15:42:47.914199 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:47.914164 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:47.914199 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:47.914200 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs97q\" (UniqueName: \"kubernetes.io/projected/255e6afe-1b4e-4306-8425-9ee44cd3b78a-kube-api-access-gs97q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:47.914421 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:47.914227 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:48.015180 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.015144 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:48.015180 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.015182 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs97q\" (UniqueName: \"kubernetes.io/projected/255e6afe-1b4e-4306-8425-9ee44cd3b78a-kube-api-access-gs97q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:48.015427 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.015207 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:48.015582 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.015561 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:48.015753 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.015650 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:48.024760 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.024737 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs97q\" (UniqueName: \"kubernetes.io/projected/255e6afe-1b4e-4306-8425-9ee44cd3b78a-kube-api-access-gs97q\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:48.128585 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.128499 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:48.261939 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.261912 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb"] Jul 24 15:42:48.263348 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:42:48.263322 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255e6afe_1b4e_4306_8425_9ee44cd3b78a.slice/crio-442e553b4b6ef00b7139566b0ab0f3ae5fc7e9ebf731a8d45056199eb931dd07 WatchSource:0}: Error finding container 442e553b4b6ef00b7139566b0ab0f3ae5fc7e9ebf731a8d45056199eb931dd07: Status 404 returned error can't find the container with id 442e553b4b6ef00b7139566b0ab0f3ae5fc7e9ebf731a8d45056199eb931dd07 Jul 24 15:42:48.422548 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.419272 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn"] Jul 24 15:42:48.423553 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.423526 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.428997 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.428972 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn"] Jul 24 15:42:48.524182 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.524143 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggt25\" (UniqueName: \"kubernetes.io/projected/6d439d65-e9d3-4c75-b566-da471930ff89-kube-api-access-ggt25\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.524363 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.524189 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.524363 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.524252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.624976 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.624935 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggt25\" (UniqueName: \"kubernetes.io/projected/6d439d65-e9d3-4c75-b566-da471930ff89-kube-api-access-ggt25\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.625176 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.624985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.625176 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.625022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.625402 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.625379 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.625467 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.625410 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.634415 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.634393 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggt25\" (UniqueName: \"kubernetes.io/projected/6d439d65-e9d3-4c75-b566-da471930ff89-kube-api-access-ggt25\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.733968 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.733892 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:48.818811 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.818783 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b"] Jul 24 15:42:48.821892 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.821873 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:48.829404 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.829377 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b"] Jul 24 15:42:48.868455 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.868433 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn"] Jul 24 15:42:48.870020 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:42:48.869990 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d439d65_e9d3_4c75_b566_da471930ff89.slice/crio-a27aed2e518b15d237775bbd584e546729f113702d62beb7637d94bf9526e8e4 WatchSource:0}: Error finding container a27aed2e518b15d237775bbd584e546729f113702d62beb7637d94bf9526e8e4: Status 404 returned error can't find the container with id a27aed2e518b15d237775bbd584e546729f113702d62beb7637d94bf9526e8e4 Jul 24 15:42:48.927797 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.927774 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpcxx\" (UniqueName: \"kubernetes.io/projected/9c874534-8528-4f24-9f55-c7f9adf9336f-kube-api-access-mpcxx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:48.927893 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.927837 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:48.927893 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:48.927868 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:49.028754 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.028721 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:49.028920 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.028769 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:49.028920 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.028895 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpcxx\" (UniqueName: \"kubernetes.io/projected/9c874534-8528-4f24-9f55-c7f9adf9336f-kube-api-access-mpcxx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:49.029143 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.029124 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:49.029197 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.029175 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:49.038585 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.038564 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpcxx\" (UniqueName: \"kubernetes.io/projected/9c874534-8528-4f24-9f55-c7f9adf9336f-kube-api-access-mpcxx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:49.135190 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.135153 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:49.150239 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.150208 2571 generic.go:356] "Generic (PLEG): container finished" podID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerID="d57f0d85d206e92dd1e920c792f317047f42ed8766fca9ba9f6b8d30ef9c6853" exitCode=0 Jul 24 15:42:49.150341 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.150282 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" event={"ID":"255e6afe-1b4e-4306-8425-9ee44cd3b78a","Type":"ContainerDied","Data":"d57f0d85d206e92dd1e920c792f317047f42ed8766fca9ba9f6b8d30ef9c6853"} Jul 24 15:42:49.150414 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.150371 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" event={"ID":"255e6afe-1b4e-4306-8425-9ee44cd3b78a","Type":"ContainerStarted","Data":"442e553b4b6ef00b7139566b0ab0f3ae5fc7e9ebf731a8d45056199eb931dd07"} Jul 24 15:42:49.173739 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.152356 2571 generic.go:356] "Generic (PLEG): container finished" podID="6d439d65-e9d3-4c75-b566-da471930ff89" containerID="16b39ca0fc2cbc60611cdbe23fae0ac830293de1fdc62025755b2b41398275ae" exitCode=0 Jul 24 15:42:49.173739 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.152476 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" event={"ID":"6d439d65-e9d3-4c75-b566-da471930ff89","Type":"ContainerDied","Data":"16b39ca0fc2cbc60611cdbe23fae0ac830293de1fdc62025755b2b41398275ae"} Jul 24 15:42:49.173739 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.152515 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" event={"ID":"6d439d65-e9d3-4c75-b566-da471930ff89","Type":"ContainerStarted","Data":"a27aed2e518b15d237775bbd584e546729f113702d62beb7637d94bf9526e8e4"} Jul 24 15:42:49.224526 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.224489 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv"] Jul 24 15:42:49.229141 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.228685 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.237623 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.237594 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv"] Jul 24 15:42:49.285395 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.285339 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b"] Jul 24 15:42:49.287464 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:42:49.287436 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c874534_8528_4f24_9f55_c7f9adf9336f.slice/crio-adaffb0af305a358e07c6309ebbba1f18f3ea1d4ef521c0093af451d32012821 WatchSource:0}: Error finding container adaffb0af305a358e07c6309ebbba1f18f3ea1d4ef521c0093af451d32012821: Status 404 returned error can't find the container with id adaffb0af305a358e07c6309ebbba1f18f3ea1d4ef521c0093af451d32012821 Jul 24 15:42:49.332344 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.332318 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxlt2\" (UniqueName: \"kubernetes.io/projected/70db67de-aa2a-4207-9418-a4ac0931d96e-kube-api-access-pxlt2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.332467 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.332364 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.332512 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.332477 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.433475 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.433440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxlt2\" (UniqueName: \"kubernetes.io/projected/70db67de-aa2a-4207-9418-a4ac0931d96e-kube-api-access-pxlt2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.433614 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.433484 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.433614 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.433532 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.433959 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.433938 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.434002 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.433951 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.442150 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.442127 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxlt2\" (UniqueName: \"kubernetes.io/projected/70db67de-aa2a-4207-9418-a4ac0931d96e-kube-api-access-pxlt2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.542915 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.542817 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:49.679586 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:49.679532 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv"] Jul 24 15:42:49.681679 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:42:49.681653 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70db67de_aa2a_4207_9418_a4ac0931d96e.slice/crio-ea18cb51b55adae0471b5291a7dd979f1b8249e294eeb152bf5a17755df31ca0 WatchSource:0}: Error finding container ea18cb51b55adae0471b5291a7dd979f1b8249e294eeb152bf5a17755df31ca0: Status 404 returned error can't find the container with id ea18cb51b55adae0471b5291a7dd979f1b8249e294eeb152bf5a17755df31ca0 Jul 24 15:42:50.159171 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:50.159139 2571 generic.go:356] "Generic (PLEG): container finished" podID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerID="56c08a9c9780cc9289bad6c78cba9806fd514c2e9b98bb84d4541b78e77e8e53" exitCode=0 Jul 24 15:42:50.159561 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:50.159207 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" event={"ID":"255e6afe-1b4e-4306-8425-9ee44cd3b78a","Type":"ContainerDied","Data":"56c08a9c9780cc9289bad6c78cba9806fd514c2e9b98bb84d4541b78e77e8e53"} Jul 24 15:42:50.160907 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:50.160884 2571 generic.go:356] "Generic (PLEG): container finished" podID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerID="eb45ee5e636c34cd6fd04d372740aa41dc42c79500b56fbf6e6fc3f8d22ed3b7" exitCode=0 Jul 24 15:42:50.161029 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:50.160948 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" event={"ID":"70db67de-aa2a-4207-9418-a4ac0931d96e","Type":"ContainerDied","Data":"eb45ee5e636c34cd6fd04d372740aa41dc42c79500b56fbf6e6fc3f8d22ed3b7"} Jul 24 15:42:50.161029 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:50.160978 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" event={"ID":"70db67de-aa2a-4207-9418-a4ac0931d96e","Type":"ContainerStarted","Data":"ea18cb51b55adae0471b5291a7dd979f1b8249e294eeb152bf5a17755df31ca0"} Jul 24 15:42:50.162464 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:50.162442 2571 generic.go:356] "Generic (PLEG): container finished" podID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerID="be4f5f0de44a8d9f8894a7b9a8c87cb878c245db3aec2c455b6277e403fa3c2b" exitCode=0 Jul 24 15:42:50.162555 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:50.162492 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" event={"ID":"9c874534-8528-4f24-9f55-c7f9adf9336f","Type":"ContainerDied","Data":"be4f5f0de44a8d9f8894a7b9a8c87cb878c245db3aec2c455b6277e403fa3c2b"} Jul 24 15:42:50.162555 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:50.162519 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" event={"ID":"9c874534-8528-4f24-9f55-c7f9adf9336f","Type":"ContainerStarted","Data":"adaffb0af305a358e07c6309ebbba1f18f3ea1d4ef521c0093af451d32012821"} Jul 24 15:42:51.170340 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.170301 2571 generic.go:356] "Generic (PLEG): container finished" podID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerID="44056b2934ec69461038f61a65732b40fed2a2d6959cb6581cd725a8803101c8" exitCode=0 Jul 24 15:42:51.170822 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.170381 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" event={"ID":"70db67de-aa2a-4207-9418-a4ac0931d96e","Type":"ContainerDied","Data":"44056b2934ec69461038f61a65732b40fed2a2d6959cb6581cd725a8803101c8"} Jul 24 15:42:51.173214 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.173191 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" event={"ID":"9c874534-8528-4f24-9f55-c7f9adf9336f","Type":"ContainerStarted","Data":"5b6f87a7ea624004235fb06aef42b72faa4cc67bbe7848f64c7ff77fdc205b50"} Jul 24 15:42:51.175515 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.175485 2571 generic.go:356] "Generic (PLEG): container finished" podID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerID="0fb2108d1ab3e9800f59add819541417ebd3a5c988758b1fab614b2701f93ec1" exitCode=0 Jul 24 15:42:51.175624 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.175588 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" event={"ID":"255e6afe-1b4e-4306-8425-9ee44cd3b78a","Type":"ContainerDied","Data":"0fb2108d1ab3e9800f59add819541417ebd3a5c988758b1fab614b2701f93ec1"} Jul 24 15:42:51.177829 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.177807 2571 generic.go:356] "Generic (PLEG): container finished" podID="6d439d65-e9d3-4c75-b566-da471930ff89" containerID="b94bbb4a261d7a58a66703f0bbb79aca9cf92533e877f8edd33ba710e5af7189" exitCode=0 Jul 24 15:42:51.177940 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.177874 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" event={"ID":"6d439d65-e9d3-4c75-b566-da471930ff89","Type":"ContainerDied","Data":"b94bbb4a261d7a58a66703f0bbb79aca9cf92533e877f8edd33ba710e5af7189"} Jul 24 15:42:51.343646 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.343610 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:51.343850 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.343680 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:51.348787 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:51.348764 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:52.187535 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.187504 2571 generic.go:356] "Generic (PLEG): container finished" podID="6d439d65-e9d3-4c75-b566-da471930ff89" containerID="2ca0da6a8957df8556a8a4c0cb1fd0b99a645a55dd1a94e88b95cc1357f8201c" exitCode=0 Jul 24 15:42:52.187930 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.187587 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" event={"ID":"6d439d65-e9d3-4c75-b566-da471930ff89","Type":"ContainerDied","Data":"2ca0da6a8957df8556a8a4c0cb1fd0b99a645a55dd1a94e88b95cc1357f8201c"} Jul 24 15:42:52.189641 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.189616 2571 generic.go:356] "Generic (PLEG): container finished" podID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerID="a28b1358caf0bcfd461c95295d39b506866bd2feaadec18291bf80fa73f4339c" exitCode=0 Jul 24 15:42:52.189768 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.189713 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" event={"ID":"70db67de-aa2a-4207-9418-a4ac0931d96e","Type":"ContainerDied","Data":"a28b1358caf0bcfd461c95295d39b506866bd2feaadec18291bf80fa73f4339c"} Jul 24 15:42:52.191448 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.191429 2571 generic.go:356] "Generic (PLEG): container finished" podID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerID="5b6f87a7ea624004235fb06aef42b72faa4cc67bbe7848f64c7ff77fdc205b50" exitCode=0 Jul 24 15:42:52.191550 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.191453 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" event={"ID":"9c874534-8528-4f24-9f55-c7f9adf9336f","Type":"ContainerDied","Data":"5b6f87a7ea624004235fb06aef42b72faa4cc67bbe7848f64c7ff77fdc205b50"} Jul 24 15:42:52.195888 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.195870 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85469664c6-pft48" Jul 24 15:42:52.251301 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.251211 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f948c498-mxsf6"] Jul 24 15:42:52.423988 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.423960 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:52.460464 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.460436 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-util\") pod \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " Jul 24 15:42:52.460635 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.460516 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-bundle\") pod \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " Jul 24 15:42:52.460635 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.460562 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gs97q\" (UniqueName: \"kubernetes.io/projected/255e6afe-1b4e-4306-8425-9ee44cd3b78a-kube-api-access-gs97q\") pod \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\" (UID: \"255e6afe-1b4e-4306-8425-9ee44cd3b78a\") " Jul 24 15:42:52.460988 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.460961 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-bundle" (OuterVolumeSpecName: "bundle") pod "255e6afe-1b4e-4306-8425-9ee44cd3b78a" (UID: "255e6afe-1b4e-4306-8425-9ee44cd3b78a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:42:52.463071 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.463047 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255e6afe-1b4e-4306-8425-9ee44cd3b78a-kube-api-access-gs97q" (OuterVolumeSpecName: "kube-api-access-gs97q") pod "255e6afe-1b4e-4306-8425-9ee44cd3b78a" (UID: "255e6afe-1b4e-4306-8425-9ee44cd3b78a"). InnerVolumeSpecName "kube-api-access-gs97q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:42:52.466521 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.466495 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-util" (OuterVolumeSpecName: "util") pod "255e6afe-1b4e-4306-8425-9ee44cd3b78a" (UID: "255e6afe-1b4e-4306-8425-9ee44cd3b78a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:42:52.562232 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.562186 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-util\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:52.562232 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.562226 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/255e6afe-1b4e-4306-8425-9ee44cd3b78a-bundle\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:52.562232 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:52.562238 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs97q\" (UniqueName: \"kubernetes.io/projected/255e6afe-1b4e-4306-8425-9ee44cd3b78a-kube-api-access-gs97q\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:53.199231 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.199195 2571 generic.go:356] "Generic (PLEG): container finished" podID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerID="d41f778a30cb30ecd2bb15bc390bd21c7c6677701d2ebc1d13a5d7f7e970b188" exitCode=0 Jul 24 15:42:53.199625 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.199282 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" event={"ID":"9c874534-8528-4f24-9f55-c7f9adf9336f","Type":"ContainerDied","Data":"d41f778a30cb30ecd2bb15bc390bd21c7c6677701d2ebc1d13a5d7f7e970b188"} Jul 24 15:42:53.201023 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.201004 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" Jul 24 15:42:53.201146 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.201049 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb" event={"ID":"255e6afe-1b4e-4306-8425-9ee44cd3b78a","Type":"ContainerDied","Data":"442e553b4b6ef00b7139566b0ab0f3ae5fc7e9ebf731a8d45056199eb931dd07"} Jul 24 15:42:53.201146 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.201083 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442e553b4b6ef00b7139566b0ab0f3ae5fc7e9ebf731a8d45056199eb931dd07" Jul 24 15:42:53.341540 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.341517 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:53.365139 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.365115 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:53.469805 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.469771 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-util\") pod \"70db67de-aa2a-4207-9418-a4ac0931d96e\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " Jul 24 15:42:53.469953 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.469827 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-bundle\") pod \"6d439d65-e9d3-4c75-b566-da471930ff89\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " Jul 24 15:42:53.469953 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.469873 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-util\") pod \"6d439d65-e9d3-4c75-b566-da471930ff89\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " Jul 24 15:42:53.469953 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.469892 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-bundle\") pod \"70db67de-aa2a-4207-9418-a4ac0931d96e\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " Jul 24 15:42:53.469953 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.469939 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxlt2\" (UniqueName: \"kubernetes.io/projected/70db67de-aa2a-4207-9418-a4ac0931d96e-kube-api-access-pxlt2\") pod \"70db67de-aa2a-4207-9418-a4ac0931d96e\" (UID: \"70db67de-aa2a-4207-9418-a4ac0931d96e\") " Jul 24 15:42:53.470139 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.469967 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggt25\" (UniqueName: \"kubernetes.io/projected/6d439d65-e9d3-4c75-b566-da471930ff89-kube-api-access-ggt25\") pod \"6d439d65-e9d3-4c75-b566-da471930ff89\" (UID: \"6d439d65-e9d3-4c75-b566-da471930ff89\") " Jul 24 15:42:53.470441 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.470410 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-bundle" (OuterVolumeSpecName: "bundle") pod "6d439d65-e9d3-4c75-b566-da471930ff89" (UID: "6d439d65-e9d3-4c75-b566-da471930ff89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:42:53.470552 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.470531 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-bundle" (OuterVolumeSpecName: "bundle") pod "70db67de-aa2a-4207-9418-a4ac0931d96e" (UID: "70db67de-aa2a-4207-9418-a4ac0931d96e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:42:53.472280 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.472256 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d439d65-e9d3-4c75-b566-da471930ff89-kube-api-access-ggt25" (OuterVolumeSpecName: "kube-api-access-ggt25") pod "6d439d65-e9d3-4c75-b566-da471930ff89" (UID: "6d439d65-e9d3-4c75-b566-da471930ff89"). InnerVolumeSpecName "kube-api-access-ggt25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:42:53.472571 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.472557 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70db67de-aa2a-4207-9418-a4ac0931d96e-kube-api-access-pxlt2" (OuterVolumeSpecName: "kube-api-access-pxlt2") pod "70db67de-aa2a-4207-9418-a4ac0931d96e" (UID: "70db67de-aa2a-4207-9418-a4ac0931d96e"). InnerVolumeSpecName "kube-api-access-pxlt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:42:53.475340 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.475321 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-util" (OuterVolumeSpecName: "util") pod "70db67de-aa2a-4207-9418-a4ac0931d96e" (UID: "70db67de-aa2a-4207-9418-a4ac0931d96e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:42:53.475742 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.475724 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-util" (OuterVolumeSpecName: "util") pod "6d439d65-e9d3-4c75-b566-da471930ff89" (UID: "6d439d65-e9d3-4c75-b566-da471930ff89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:42:53.570863 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.570829 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-util\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:53.570863 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.570857 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-bundle\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:53.570863 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.570866 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6d439d65-e9d3-4c75-b566-da471930ff89-util\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:53.571070 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.570874 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70db67de-aa2a-4207-9418-a4ac0931d96e-bundle\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:53.571070 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.570883 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxlt2\" (UniqueName: \"kubernetes.io/projected/70db67de-aa2a-4207-9418-a4ac0931d96e-kube-api-access-pxlt2\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:53.571070 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:53.570891 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggt25\" (UniqueName: \"kubernetes.io/projected/6d439d65-e9d3-4c75-b566-da471930ff89-kube-api-access-ggt25\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:54.208265 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.208238 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" Jul 24 15:42:54.208731 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.208280 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv" event={"ID":"70db67de-aa2a-4207-9418-a4ac0931d96e","Type":"ContainerDied","Data":"ea18cb51b55adae0471b5291a7dd979f1b8249e294eeb152bf5a17755df31ca0"} Jul 24 15:42:54.208731 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.208337 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea18cb51b55adae0471b5291a7dd979f1b8249e294eeb152bf5a17755df31ca0" Jul 24 15:42:54.210316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.210285 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" event={"ID":"6d439d65-e9d3-4c75-b566-da471930ff89","Type":"ContainerDied","Data":"a27aed2e518b15d237775bbd584e546729f113702d62beb7637d94bf9526e8e4"} Jul 24 15:42:54.210316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.210314 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn" Jul 24 15:42:54.210514 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.210324 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a27aed2e518b15d237775bbd584e546729f113702d62beb7637d94bf9526e8e4" Jul 24 15:42:54.345096 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.345070 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:54.478350 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.478319 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpcxx\" (UniqueName: \"kubernetes.io/projected/9c874534-8528-4f24-9f55-c7f9adf9336f-kube-api-access-mpcxx\") pod \"9c874534-8528-4f24-9f55-c7f9adf9336f\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " Jul 24 15:42:54.478532 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.478385 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-bundle\") pod \"9c874534-8528-4f24-9f55-c7f9adf9336f\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " Jul 24 15:42:54.478532 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.478422 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-util\") pod \"9c874534-8528-4f24-9f55-c7f9adf9336f\" (UID: \"9c874534-8528-4f24-9f55-c7f9adf9336f\") " Jul 24 15:42:54.478917 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.478892 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-bundle" (OuterVolumeSpecName: "bundle") pod "9c874534-8528-4f24-9f55-c7f9adf9336f" (UID: "9c874534-8528-4f24-9f55-c7f9adf9336f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:42:54.480751 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.480722 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c874534-8528-4f24-9f55-c7f9adf9336f-kube-api-access-mpcxx" (OuterVolumeSpecName: "kube-api-access-mpcxx") pod "9c874534-8528-4f24-9f55-c7f9adf9336f" (UID: "9c874534-8528-4f24-9f55-c7f9adf9336f"). InnerVolumeSpecName "kube-api-access-mpcxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:42:54.483972 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.483939 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-util" (OuterVolumeSpecName: "util") pod "9c874534-8528-4f24-9f55-c7f9adf9336f" (UID: "9c874534-8528-4f24-9f55-c7f9adf9336f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:42:54.579443 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.579362 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpcxx\" (UniqueName: \"kubernetes.io/projected/9c874534-8528-4f24-9f55-c7f9adf9336f-kube-api-access-mpcxx\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:54.579443 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.579392 2571 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-bundle\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:54.579443 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:54.579400 2571 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c874534-8528-4f24-9f55-c7f9adf9336f-util\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:42:55.216944 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:55.216915 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" Jul 24 15:42:55.216944 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:55.216930 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b" event={"ID":"9c874534-8528-4f24-9f55-c7f9adf9336f","Type":"ContainerDied","Data":"adaffb0af305a358e07c6309ebbba1f18f3ea1d4ef521c0093af451d32012821"} Jul 24 15:42:55.217370 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:42:55.216971 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adaffb0af305a358e07c6309ebbba1f18f3ea1d4ef521c0093af451d32012821" Jul 24 15:43:13.700248 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700207 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8"] Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700618 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerName="pull" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700634 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerName="pull" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700646 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d439d65-e9d3-4c75-b566-da471930ff89" containerName="util" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700651 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d439d65-e9d3-4c75-b566-da471930ff89" containerName="util" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700664 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerName="util" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700669 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerName="util" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700675 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerName="pull" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700679 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerName="pull" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700685 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerName="pull" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700690 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerName="pull" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700714 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d439d65-e9d3-4c75-b566-da471930ff89" containerName="pull" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700720 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d439d65-e9d3-4c75-b566-da471930ff89" containerName="pull" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700727 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerName="extract" Jul 24 15:43:13.700724 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700732 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700739 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700744 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700749 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d439d65-e9d3-4c75-b566-da471930ff89" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700754 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d439d65-e9d3-4c75-b566-da471930ff89" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700765 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerName="util" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700772 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerName="util" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700778 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerName="util" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700782 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerName="util" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700788 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700793 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700843 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c874534-8528-4f24-9f55-c7f9adf9336f" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700851 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="255e6afe-1b4e-4306-8425-9ee44cd3b78a" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700859 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="70db67de-aa2a-4207-9418-a4ac0931d96e" containerName="extract" Jul 24 15:43:13.701186 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.700865 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d439d65-e9d3-4c75-b566-da471930ff89" containerName="extract" Jul 24 15:43:13.705369 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.705354 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" Jul 24 15:43:13.708173 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.708155 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 15:43:13.708328 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.708157 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:43:13.709068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.709050 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-cpp9b\"" Jul 24 15:43:13.709181 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.709052 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:43:13.713139 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.713081 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8"] Jul 24 15:43:13.856192 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.856159 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz9qj\" (UniqueName: \"kubernetes.io/projected/9a44395c-e5fe-4f96-82a2-725abc821320-kube-api-access-tz9qj\") pod \"dns-operator-controller-manager-864dd78c4c-wn4j8\" (UID: \"9a44395c-e5fe-4f96-82a2-725abc821320\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" Jul 24 15:43:13.957550 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.957456 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz9qj\" (UniqueName: \"kubernetes.io/projected/9a44395c-e5fe-4f96-82a2-725abc821320-kube-api-access-tz9qj\") pod \"dns-operator-controller-manager-864dd78c4c-wn4j8\" (UID: \"9a44395c-e5fe-4f96-82a2-725abc821320\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" Jul 24 15:43:13.966787 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:13.966752 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz9qj\" (UniqueName: \"kubernetes.io/projected/9a44395c-e5fe-4f96-82a2-725abc821320-kube-api-access-tz9qj\") pod \"dns-operator-controller-manager-864dd78c4c-wn4j8\" (UID: \"9a44395c-e5fe-4f96-82a2-725abc821320\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" Jul 24 15:43:14.017878 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.017840 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" Jul 24 15:43:14.166213 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.166188 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8"] Jul 24 15:43:14.168519 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:43:14.168486 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a44395c_e5fe_4f96_82a2_725abc821320.slice/crio-10a8e4e6b5a279884d75640f7fd03daf0556c46eb0e3f6473492a517fc0b372e WatchSource:0}: Error finding container 10a8e4e6b5a279884d75640f7fd03daf0556c46eb0e3f6473492a517fc0b372e: Status 404 returned error can't find the container with id 10a8e4e6b5a279884d75640f7fd03daf0556c46eb0e3f6473492a517fc0b372e Jul 24 15:43:14.310716 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.310656 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" event={"ID":"9a44395c-e5fe-4f96-82a2-725abc821320","Type":"ContainerStarted","Data":"10a8e4e6b5a279884d75640f7fd03daf0556c46eb0e3f6473492a517fc0b372e"} Jul 24 15:43:14.775478 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.775447 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7"] Jul 24 15:43:14.778880 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.778860 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:14.781495 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.781472 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 15:43:14.781738 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.781723 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 15:43:14.781843 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.781796 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9ds7q\"" Jul 24 15:43:14.790936 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.790911 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7"] Jul 24 15:43:14.865182 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.865149 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07dc3a68-a3b7-48c2-8df7-9195a8a831ee-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dzcn7\" (UID: \"07dc3a68-a3b7-48c2-8df7-9195a8a831ee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:14.865322 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.865216 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/07dc3a68-a3b7-48c2-8df7-9195a8a831ee-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dzcn7\" (UID: \"07dc3a68-a3b7-48c2-8df7-9195a8a831ee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:14.865322 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.865235 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxp9h\" (UniqueName: \"kubernetes.io/projected/07dc3a68-a3b7-48c2-8df7-9195a8a831ee-kube-api-access-dxp9h\") pod \"kuadrant-console-plugin-66b5774985-dzcn7\" (UID: \"07dc3a68-a3b7-48c2-8df7-9195a8a831ee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:14.966710 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.966653 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/07dc3a68-a3b7-48c2-8df7-9195a8a831ee-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dzcn7\" (UID: \"07dc3a68-a3b7-48c2-8df7-9195a8a831ee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:14.966710 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.966710 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxp9h\" (UniqueName: \"kubernetes.io/projected/07dc3a68-a3b7-48c2-8df7-9195a8a831ee-kube-api-access-dxp9h\") pod \"kuadrant-console-plugin-66b5774985-dzcn7\" (UID: \"07dc3a68-a3b7-48c2-8df7-9195a8a831ee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:14.966933 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.966793 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07dc3a68-a3b7-48c2-8df7-9195a8a831ee-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dzcn7\" (UID: \"07dc3a68-a3b7-48c2-8df7-9195a8a831ee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:14.967492 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.967470 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/07dc3a68-a3b7-48c2-8df7-9195a8a831ee-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-dzcn7\" (UID: \"07dc3a68-a3b7-48c2-8df7-9195a8a831ee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:14.969426 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.969405 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/07dc3a68-a3b7-48c2-8df7-9195a8a831ee-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-dzcn7\" (UID: \"07dc3a68-a3b7-48c2-8df7-9195a8a831ee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:14.978318 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:14.978289 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxp9h\" (UniqueName: \"kubernetes.io/projected/07dc3a68-a3b7-48c2-8df7-9195a8a831ee-kube-api-access-dxp9h\") pod \"kuadrant-console-plugin-66b5774985-dzcn7\" (UID: \"07dc3a68-a3b7-48c2-8df7-9195a8a831ee\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:15.090520 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:15.090443 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" Jul 24 15:43:15.552680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:15.552610 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7"] Jul 24 15:43:15.555218 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:43:15.555196 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07dc3a68_a3b7_48c2_8df7_9195a8a831ee.slice/crio-93b9d5f20705b46a5d26e50848c80864b0a31661450088faf0ee185cd01636af WatchSource:0}: Error finding container 93b9d5f20705b46a5d26e50848c80864b0a31661450088faf0ee185cd01636af: Status 404 returned error can't find the container with id 93b9d5f20705b46a5d26e50848c80864b0a31661450088faf0ee185cd01636af Jul 24 15:43:16.328542 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:16.328496 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" event={"ID":"07dc3a68-a3b7-48c2-8df7-9195a8a831ee","Type":"ContainerStarted","Data":"93b9d5f20705b46a5d26e50848c80864b0a31661450088faf0ee185cd01636af"} Jul 24 15:43:16.330824 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:16.330795 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" event={"ID":"9a44395c-e5fe-4f96-82a2-725abc821320","Type":"ContainerStarted","Data":"0010cfff73c6ce1517537cf4057c094536711bd465b663cbb0bc7ed6464b5be0"} Jul 24 15:43:16.331238 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:16.331190 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" Jul 24 15:43:16.353159 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:16.353097 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" podStartSLOduration=1.441601993 podStartE2EDuration="3.353076064s" totalImagesPullingTime="1.911474071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:43:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:43:16.34975443 +0000 UTC m=+437.582482105" watchObservedRunningTime="2026-07-24 15:43:16.353076064 +0000 UTC m=+437.585803743" Jul 24 15:43:18.227780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:18.227734 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77f948c498-mxsf6" podUID="9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" containerName="console" containerID="cri-o://478bf5449d0acbc911c49e73463713bca60816303f5b9a1bc83658acc998b2e2" gracePeriod=15 Jul 24 15:43:19.351492 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:19.351301 2571 generic.go:356] "Generic (PLEG): container finished" podID="9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" containerID="478bf5449d0acbc911c49e73463713bca60816303f5b9a1bc83658acc998b2e2" exitCode=0 Jul 24 15:43:19.351492 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:19.351388 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f948c498-mxsf6" event={"ID":"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b","Type":"ContainerDied","Data":"478bf5449d0acbc911c49e73463713bca60816303f5b9a1bc83658acc998b2e2"} Jul 24 15:43:20.271145 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.271123 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:43:20.366384 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.366340 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f948c498-mxsf6" event={"ID":"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b","Type":"ContainerDied","Data":"f9aa6fef1cf8ccde442866c7e671d5112e95efd729903f6ef32f7af8c9a0967c"} Jul 24 15:43:20.366384 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.366380 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f948c498-mxsf6" Jul 24 15:43:20.366906 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.366425 2571 scope.go:117] "RemoveContainer" containerID="478bf5449d0acbc911c49e73463713bca60816303f5b9a1bc83658acc998b2e2" Jul 24 15:43:20.419538 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.419509 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-config\") pod \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " Jul 24 15:43:20.419743 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.419546 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-trusted-ca-bundle\") pod \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " Jul 24 15:43:20.419743 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.419604 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-oauth-serving-cert\") pod \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " Jul 24 15:43:20.419743 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.419670 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-oauth-config\") pod \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " Jul 24 15:43:20.419743 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.419731 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4bw4\" (UniqueName: \"kubernetes.io/projected/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-kube-api-access-n4bw4\") pod \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " Jul 24 15:43:20.419947 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.419764 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-serving-cert\") pod \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " Jul 24 15:43:20.419947 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.419827 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-service-ca\") pod \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\" (UID: \"9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b\") " Jul 24 15:43:20.420040 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.420007 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" (UID: "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:20.420040 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.420005 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-config" (OuterVolumeSpecName: "console-config") pod "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" (UID: "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:20.420135 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.420059 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" (UID: "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:20.420135 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.420101 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-config\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:43:20.420135 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.420119 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-trusted-ca-bundle\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:43:20.420310 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.420286 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-service-ca" (OuterVolumeSpecName: "service-ca") pod "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" (UID: "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:20.422071 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.422044 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" (UID: "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:20.422184 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.422062 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-kube-api-access-n4bw4" (OuterVolumeSpecName: "kube-api-access-n4bw4") pod "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" (UID: "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b"). InnerVolumeSpecName "kube-api-access-n4bw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:43:20.422184 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.422146 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" (UID: "9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:20.520628 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.520542 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:43:20.520628 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.520575 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-oauth-serving-cert\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:43:20.520628 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.520586 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-oauth-config\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:43:20.520628 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.520595 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bw4\" (UniqueName: \"kubernetes.io/projected/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-kube-api-access-n4bw4\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:43:20.520628 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.520604 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b-console-serving-cert\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:43:20.691493 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.691439 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f948c498-mxsf6"] Jul 24 15:43:20.695122 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:20.695090 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77f948c498-mxsf6"] Jul 24 15:43:21.252739 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:21.252683 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" path="/var/lib/kubelet/pods/9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b/volumes" Jul 24 15:43:27.341561 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:27.341535 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-wn4j8" Jul 24 15:43:39.481411 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:39.481370 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" event={"ID":"07dc3a68-a3b7-48c2-8df7-9195a8a831ee","Type":"ContainerStarted","Data":"1e0f1a433a4aa7093e8dc6cf6ed59896478f2c331ca52771de15c182c932d760"} Jul 24 15:43:39.498551 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:39.498498 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-dzcn7" podStartSLOduration=2.263022008 podStartE2EDuration="25.498483078s" totalImagesPullingTime="23.23546107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:43:39.497288917 +0000 UTC m=+460.730016596" watchObservedRunningTime="2026-07-24 15:43:39.498483078 +0000 UTC m=+460.731210756" Jul 24 15:43:56.234271 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.234237 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-ltbdk"] Jul 24 15:43:56.234750 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.234594 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" containerName="console" Jul 24 15:43:56.234750 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.234606 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" containerName="console" Jul 24 15:43:56.234750 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.234672 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bd70a16-04c1-4813-86cd-ae7a7e3b8d2b" containerName="console" Jul 24 15:43:56.237235 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.237220 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ltbdk" Jul 24 15:43:56.239638 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.239623 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-bkphj\"" Jul 24 15:43:56.244967 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.244945 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ltbdk"] Jul 24 15:43:56.335756 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.335719 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djfvl\" (UniqueName: \"kubernetes.io/projected/84132b1b-0a44-436c-b0df-f2b13501a74f-kube-api-access-djfvl\") pod \"authorino-58cd955675-ltbdk\" (UID: \"84132b1b-0a44-436c-b0df-f2b13501a74f\") " pod="kuadrant-system/authorino-58cd955675-ltbdk" Jul 24 15:43:56.437093 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.437057 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djfvl\" (UniqueName: \"kubernetes.io/projected/84132b1b-0a44-436c-b0df-f2b13501a74f-kube-api-access-djfvl\") pod \"authorino-58cd955675-ltbdk\" (UID: \"84132b1b-0a44-436c-b0df-f2b13501a74f\") " pod="kuadrant-system/authorino-58cd955675-ltbdk" Jul 24 15:43:56.445928 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.445899 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djfvl\" (UniqueName: \"kubernetes.io/projected/84132b1b-0a44-436c-b0df-f2b13501a74f-kube-api-access-djfvl\") pod \"authorino-58cd955675-ltbdk\" (UID: \"84132b1b-0a44-436c-b0df-f2b13501a74f\") " pod="kuadrant-system/authorino-58cd955675-ltbdk" Jul 24 15:43:56.546640 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.546607 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ltbdk" Jul 24 15:43:56.684664 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:56.684632 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ltbdk"] Jul 24 15:43:56.686202 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:43:56.686180 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84132b1b_0a44_436c_b0df_f2b13501a74f.slice/crio-bb39c26ca9d020dcc2246fd4b10492b55e861bf03d3f4d998bb813cc01202f0b WatchSource:0}: Error finding container bb39c26ca9d020dcc2246fd4b10492b55e861bf03d3f4d998bb813cc01202f0b: Status 404 returned error can't find the container with id bb39c26ca9d020dcc2246fd4b10492b55e861bf03d3f4d998bb813cc01202f0b Jul 24 15:43:57.583156 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:43:57.583095 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ltbdk" event={"ID":"84132b1b-0a44-436c-b0df-f2b13501a74f","Type":"ContainerStarted","Data":"bb39c26ca9d020dcc2246fd4b10492b55e861bf03d3f4d998bb813cc01202f0b"} Jul 24 15:44:00.601631 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:00.601597 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ltbdk" event={"ID":"84132b1b-0a44-436c-b0df-f2b13501a74f","Type":"ContainerStarted","Data":"38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e"} Jul 24 15:44:00.617299 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:00.617251 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-ltbdk" podStartSLOduration=1.597233497 podStartE2EDuration="4.617236158s" totalImagesPullingTime="3.020002661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:43:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:00.615265844 +0000 UTC m=+481.847993523" watchObservedRunningTime="2026-07-24 15:44:00.617236158 +0000 UTC m=+481.849963836" Jul 24 15:44:01.083263 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:01.083224 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ltbdk"] Jul 24 15:44:02.611325 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:02.611258 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-ltbdk" podUID="84132b1b-0a44-436c-b0df-f2b13501a74f" containerName="authorino" containerID="cri-o://38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e" gracePeriod=30 Jul 24 15:44:02.857858 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:02.857837 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ltbdk" Jul 24 15:44:02.994455 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:02.994422 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djfvl\" (UniqueName: \"kubernetes.io/projected/84132b1b-0a44-436c-b0df-f2b13501a74f-kube-api-access-djfvl\") pod \"84132b1b-0a44-436c-b0df-f2b13501a74f\" (UID: \"84132b1b-0a44-436c-b0df-f2b13501a74f\") " Jul 24 15:44:02.996659 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:02.996636 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84132b1b-0a44-436c-b0df-f2b13501a74f-kube-api-access-djfvl" (OuterVolumeSpecName: "kube-api-access-djfvl") pod "84132b1b-0a44-436c-b0df-f2b13501a74f" (UID: "84132b1b-0a44-436c-b0df-f2b13501a74f"). InnerVolumeSpecName "kube-api-access-djfvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:44:03.095377 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.095342 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djfvl\" (UniqueName: \"kubernetes.io/projected/84132b1b-0a44-436c-b0df-f2b13501a74f-kube-api-access-djfvl\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:44:03.618051 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.617958 2571 generic.go:356] "Generic (PLEG): container finished" podID="84132b1b-0a44-436c-b0df-f2b13501a74f" containerID="38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e" exitCode=0 Jul 24 15:44:03.618051 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.617996 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ltbdk" event={"ID":"84132b1b-0a44-436c-b0df-f2b13501a74f","Type":"ContainerDied","Data":"38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e"} Jul 24 15:44:03.618051 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.618007 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ltbdk" Jul 24 15:44:03.618051 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.618025 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ltbdk" event={"ID":"84132b1b-0a44-436c-b0df-f2b13501a74f","Type":"ContainerDied","Data":"bb39c26ca9d020dcc2246fd4b10492b55e861bf03d3f4d998bb813cc01202f0b"} Jul 24 15:44:03.618051 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.618041 2571 scope.go:117] "RemoveContainer" containerID="38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e" Jul 24 15:44:03.629097 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.629075 2571 scope.go:117] "RemoveContainer" containerID="38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e" Jul 24 15:44:03.629360 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:44:03.629340 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e\": container with ID starting with 38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e not found: ID does not exist" containerID="38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e" Jul 24 15:44:03.629420 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.629368 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e"} err="failed to get container status \"38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e\": rpc error: code = NotFound desc = could not find container \"38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e\": container with ID starting with 38eac083a9e265166a5114828afb7b470e5bf911a807c31649c38244e0bb032e not found: ID does not exist" Jul 24 15:44:03.636517 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.636493 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ltbdk"] Jul 24 15:44:03.643224 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:03.643203 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-ltbdk"] Jul 24 15:44:05.251366 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:05.251329 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84132b1b-0a44-436c-b0df-f2b13501a74f" path="/var/lib/kubelet/pods/84132b1b-0a44-436c-b0df-f2b13501a74f/volumes" Jul 24 15:44:31.225324 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.225293 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-67bb47cc5d-c6fhr"] Jul 24 15:44:31.225800 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.225610 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84132b1b-0a44-436c-b0df-f2b13501a74f" containerName="authorino" Jul 24 15:44:31.225800 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.225620 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="84132b1b-0a44-436c-b0df-f2b13501a74f" containerName="authorino" Jul 24 15:44:31.225800 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.225676 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="84132b1b-0a44-436c-b0df-f2b13501a74f" containerName="authorino" Jul 24 15:44:31.231729 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.231687 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:44:31.234835 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.234806 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 15:44:31.234965 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.234868 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-g5tpr\"" Jul 24 15:44:31.234965 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.234815 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 15:44:31.251606 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.251576 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67bb47cc5d-c6fhr"] Jul 24 15:44:31.321257 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.321215 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-webhook-cert\") pod \"maas-controller-67bb47cc5d-c6fhr\" (UID: \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\") " pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:44:31.321461 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.321360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nppfm\" (UniqueName: \"kubernetes.io/projected/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-kube-api-access-nppfm\") pod \"maas-controller-67bb47cc5d-c6fhr\" (UID: \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\") " pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:44:31.422680 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.422655 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nppfm\" (UniqueName: \"kubernetes.io/projected/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-kube-api-access-nppfm\") pod \"maas-controller-67bb47cc5d-c6fhr\" (UID: \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\") " pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:44:31.422859 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.422748 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-webhook-cert\") pod \"maas-controller-67bb47cc5d-c6fhr\" (UID: \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\") " pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:44:31.425178 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.425151 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-webhook-cert\") pod \"maas-controller-67bb47cc5d-c6fhr\" (UID: \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\") " pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:44:31.430966 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.430950 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nppfm\" (UniqueName: \"kubernetes.io/projected/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-kube-api-access-nppfm\") pod \"maas-controller-67bb47cc5d-c6fhr\" (UID: \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\") " pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:44:31.542438 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.542398 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:44:31.679119 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.679094 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-67bb47cc5d-c6fhr"] Jul 24 15:44:31.680967 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:44:31.680940 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3cfc5b_a93e_4384_8a93_6e351aa0f886.slice/crio-8d591b30ed132a454652abb1d60f137f9ed2e97c7382973599d593b1406fdfc0 WatchSource:0}: Error finding container 8d591b30ed132a454652abb1d60f137f9ed2e97c7382973599d593b1406fdfc0: Status 404 returned error can't find the container with id 8d591b30ed132a454652abb1d60f137f9ed2e97c7382973599d593b1406fdfc0 Jul 24 15:44:31.759297 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:31.759260 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" event={"ID":"fc3cfc5b-a93e-4384-8a93-6e351aa0f886","Type":"ContainerStarted","Data":"8d591b30ed132a454652abb1d60f137f9ed2e97c7382973599d593b1406fdfc0"} Jul 24 15:44:34.778782 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:34.778749 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" event={"ID":"fc3cfc5b-a93e-4384-8a93-6e351aa0f886","Type":"ContainerStarted","Data":"6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6"} Jul 24 15:44:34.779180 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:34.778831 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:44:34.800273 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:34.800217 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" podStartSLOduration=1.496361248 podStartE2EDuration="3.800202179s" totalImagesPullingTime="2.303840931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:44:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:34.798106613 +0000 UTC m=+516.030834344" watchObservedRunningTime="2026-07-24 15:44:34.800202179 +0000 UTC m=+516.032929888" Jul 24 15:44:45.785837 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:44:45.785804 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:45:10.247863 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.247830 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54"] Jul 24 15:45:10.255076 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.255057 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.259235 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.259206 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 15:45:10.259357 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.259215 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:45:10.260257 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.260236 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bjdfq\"" Jul 24 15:45:10.260352 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.260243 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 15:45:10.270418 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.270393 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54"] Jul 24 15:45:10.346170 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.346138 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-maas-api-tls\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.346170 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.346172 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hflb\" (UniqueName: \"kubernetes.io/projected/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-kube-api-access-4hflb\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.346403 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.346213 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-openshift-service-ca\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.447488 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.447451 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-openshift-service-ca\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.447659 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.447521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-maas-api-tls\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.447659 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.447544 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hflb\" (UniqueName: \"kubernetes.io/projected/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-kube-api-access-4hflb\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.447659 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:45:10.447633 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 24 15:45:10.447807 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:45:10.447734 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-maas-api-tls podName:af843a3a-82ec-40ad-8b94-e58e2b3c3e9f nodeName:}" failed. No retries permitted until 2026-07-24 15:45:10.947713298 +0000 UTC m=+552.180440955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-maas-api-tls") pod "maas-api-79b8cf8fb9-bfx54" (UID: "af843a3a-82ec-40ad-8b94-e58e2b3c3e9f") : secret "maas-api-serving-cert" not found Jul 24 15:45:10.448229 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.448209 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-openshift-service-ca\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.461348 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.461327 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hflb\" (UniqueName: \"kubernetes.io/projected/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-kube-api-access-4hflb\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.754710 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.754664 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk"] Jul 24 15:45:10.758061 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.758046 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:10.760550 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.760522 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kgmvx\"" Jul 24 15:45:10.760654 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.760582 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 15:45:10.767264 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.767244 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk"] Jul 24 15:45:10.850856 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.850812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncjkj\" (UniqueName: \"kubernetes.io/projected/eb53ba3b-7c64-4204-927d-d99794ea3c73-kube-api-access-ncjkj\") pod \"payload-pre-processing-687dbb6cf6-55mrk\" (UID: \"eb53ba3b-7c64-4204-927d-d99794ea3c73\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:10.851036 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.850962 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb53ba3b-7c64-4204-927d-d99794ea3c73-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-55mrk\" (UID: \"eb53ba3b-7c64-4204-927d-d99794ea3c73\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:10.952019 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.951988 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncjkj\" (UniqueName: \"kubernetes.io/projected/eb53ba3b-7c64-4204-927d-d99794ea3c73-kube-api-access-ncjkj\") pod \"payload-pre-processing-687dbb6cf6-55mrk\" (UID: \"eb53ba3b-7c64-4204-927d-d99794ea3c73\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:10.952201 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.952054 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-maas-api-tls\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.952201 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.952125 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb53ba3b-7c64-4204-927d-d99794ea3c73-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-55mrk\" (UID: \"eb53ba3b-7c64-4204-927d-d99794ea3c73\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:10.952922 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.952839 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb53ba3b-7c64-4204-927d-d99794ea3c73-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-55mrk\" (UID: \"eb53ba3b-7c64-4204-927d-d99794ea3c73\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:10.954712 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.954676 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/af843a3a-82ec-40ad-8b94-e58e2b3c3e9f-maas-api-tls\") pod \"maas-api-79b8cf8fb9-bfx54\" (UID: \"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f\") " pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:10.960446 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:10.960423 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncjkj\" (UniqueName: \"kubernetes.io/projected/eb53ba3b-7c64-4204-927d-d99794ea3c73-kube-api-access-ncjkj\") pod \"payload-pre-processing-687dbb6cf6-55mrk\" (UID: \"eb53ba3b-7c64-4204-927d-d99794ea3c73\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:11.070659 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:11.070573 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:11.166083 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:11.166056 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:11.208565 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:11.208526 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk"] Jul 24 15:45:11.210851 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:45:11.210807 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb53ba3b_7c64_4204_927d_d99794ea3c73.slice/crio-49e42e91ca0964c1e7bc53bfda76d59356b3cae971f1530a291eb18b935adee8 WatchSource:0}: Error finding container 49e42e91ca0964c1e7bc53bfda76d59356b3cae971f1530a291eb18b935adee8: Status 404 returned error can't find the container with id 49e42e91ca0964c1e7bc53bfda76d59356b3cae971f1530a291eb18b935adee8 Jul 24 15:45:11.305814 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:11.305792 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54"] Jul 24 15:45:11.307506 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:45:11.307480 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf843a3a_82ec_40ad_8b94_e58e2b3c3e9f.slice/crio-f8a27f064db0d7887d77199de92705462ee4d13cc22c9e49595a0074610a5382 WatchSource:0}: Error finding container f8a27f064db0d7887d77199de92705462ee4d13cc22c9e49595a0074610a5382: Status 404 returned error can't find the container with id f8a27f064db0d7887d77199de92705462ee4d13cc22c9e49595a0074610a5382 Jul 24 15:45:11.966951 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:11.966915 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" event={"ID":"eb53ba3b-7c64-4204-927d-d99794ea3c73","Type":"ContainerStarted","Data":"49e42e91ca0964c1e7bc53bfda76d59356b3cae971f1530a291eb18b935adee8"} Jul 24 15:45:11.968052 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:11.968030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" event={"ID":"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f","Type":"ContainerStarted","Data":"f8a27f064db0d7887d77199de92705462ee4d13cc22c9e49595a0074610a5382"} Jul 24 15:45:13.981872 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:13.981773 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" event={"ID":"eb53ba3b-7c64-4204-927d-d99794ea3c73","Type":"ContainerStarted","Data":"ebaa6b4147c23564384688856d9892afe26f65626bdd28532ac2a3aded18deab"} Jul 24 15:45:13.981872 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:13.981864 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:13.983521 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:13.983499 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" event={"ID":"af843a3a-82ec-40ad-8b94-e58e2b3c3e9f","Type":"ContainerStarted","Data":"e105249f56ae008b69ec7d3d9122ac4b954ac7cfe15d4d925e12d432cdf40520"} Jul 24 15:45:13.983634 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:13.983546 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:13.999116 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:13.999053 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" podStartSLOduration=1.6958416490000001 podStartE2EDuration="3.999033039s" totalImagesPullingTime="2.30319139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:13.996540393 +0000 UTC m=+555.229268051" watchObservedRunningTime="2026-07-24 15:45:13.999033039 +0000 UTC m=+555.231760718" Jul 24 15:45:14.013359 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:14.013302 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" podStartSLOduration=1.807385831 podStartE2EDuration="4.013287493s" totalImagesPullingTime="2.205901662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:14.012103342 +0000 UTC m=+555.244831020" watchObservedRunningTime="2026-07-24 15:45:14.013287493 +0000 UTC m=+555.246015171" Jul 24 15:45:19.994166 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:19.994129 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79b8cf8fb9-bfx54" Jul 24 15:45:24.991204 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:24.991169 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-55mrk" Jul 24 15:45:44.186104 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.186055 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz"] Jul 24 15:45:44.191945 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.191920 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.194481 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.194454 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mjhhv\"" Jul 24 15:45:44.194712 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.194454 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 15:45:44.195401 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.195383 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 15:45:44.195500 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.195453 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 15:45:44.201161 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.201140 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz"] Jul 24 15:45:44.245874 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.245846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6642875-2374-4c1e-aea2-816cdbbd464f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.246049 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.245880 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.246049 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.245898 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.246049 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.245976 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.246049 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.246015 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvqx8\" (UniqueName: \"kubernetes.io/projected/c6642875-2374-4c1e-aea2-816cdbbd464f-kube-api-access-pvqx8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.246195 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.246056 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.346976 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.346946 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6642875-2374-4c1e-aea2-816cdbbd464f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.346976 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.346985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.347233 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.347002 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.347233 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.347049 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.347233 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.347078 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvqx8\" (UniqueName: \"kubernetes.io/projected/c6642875-2374-4c1e-aea2-816cdbbd464f-kube-api-access-pvqx8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.347233 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.347142 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.347422 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.347383 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.347496 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.347455 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.347552 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.347528 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.349507 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.349485 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6642875-2374-4c1e-aea2-816cdbbd464f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.349677 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.349661 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6642875-2374-4c1e-aea2-816cdbbd464f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.355401 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.355380 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvqx8\" (UniqueName: \"kubernetes.io/projected/c6642875-2374-4c1e-aea2-816cdbbd464f-kube-api-access-pvqx8\") pod \"e2e-distinct-simulated-kserve-59975d69f9-rzktz\" (UID: \"c6642875-2374-4c1e-aea2-816cdbbd464f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.503100 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.503064 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:44.649435 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:44.649408 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz"] Jul 24 15:45:44.651424 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:45:44.651391 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6642875_2374_4c1e_aea2_816cdbbd464f.slice/crio-1a727e2e32768b2058b9c4c90d76e0172ed8484152fe1d513473c00e06b8f7fa WatchSource:0}: Error finding container 1a727e2e32768b2058b9c4c90d76e0172ed8484152fe1d513473c00e06b8f7fa: Status 404 returned error can't find the container with id 1a727e2e32768b2058b9c4c90d76e0172ed8484152fe1d513473c00e06b8f7fa Jul 24 15:45:45.147128 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:45.147079 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" event={"ID":"c6642875-2374-4c1e-aea2-816cdbbd464f","Type":"ContainerStarted","Data":"1a727e2e32768b2058b9c4c90d76e0172ed8484152fe1d513473c00e06b8f7fa"} Jul 24 15:45:46.153869 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:46.153833 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" event={"ID":"c6642875-2374-4c1e-aea2-816cdbbd464f","Type":"ContainerStarted","Data":"fd2c8435e02ee2a21c9f6ddcd93ed8d93c0df237ea777e090c2127f2a82f3566"} Jul 24 15:45:46.171155 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:46.171100 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" podStartSLOduration=1.1456263070000001 podStartE2EDuration="2.17108361s" totalImagesPullingTime="1.025457303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:46.170151179 +0000 UTC m=+587.402878859" watchObservedRunningTime="2026-07-24 15:45:46.17108361 +0000 UTC m=+587.403811288" Jul 24 15:45:54.503291 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:54.503248 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:54.503720 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:54.503314 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:54.515905 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:54.515880 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:55.210409 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:55.210383 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-rzktz" Jul 24 15:45:59.168139 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:59.168110 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:45:59.168639 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:59.168288 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:45:59.173647 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:59.173628 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:45:59.173810 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:45:59.173649 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:46:11.302357 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.302329 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s"] Jul 24 15:46:11.305435 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.305415 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.308261 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.308244 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 15:46:11.320609 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.320586 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s"] Jul 24 15:46:11.382147 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.382118 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ltm\" (UniqueName: \"kubernetes.io/projected/b3970d43-ba51-465c-8327-093a4542addc-kube-api-access-b6ltm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.382310 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.382157 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.382310 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.382185 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3970d43-ba51-465c-8327-093a4542addc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.382310 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.382241 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.382310 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.382286 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.382440 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.382314 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.482760 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.482719 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.482760 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.482767 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3970d43-ba51-465c-8327-093a4542addc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.483004 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.482794 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.483004 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.482826 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.483004 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.482851 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.483004 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.482893 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ltm\" (UniqueName: \"kubernetes.io/projected/b3970d43-ba51-465c-8327-093a4542addc-kube-api-access-b6ltm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.483202 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.483129 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.483202 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.483170 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.483289 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.483271 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.485265 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.485245 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3970d43-ba51-465c-8327-093a4542addc-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.485426 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.485410 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3970d43-ba51-465c-8327-093a4542addc-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.492780 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.492759 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ltm\" (UniqueName: \"kubernetes.io/projected/b3970d43-ba51-465c-8327-093a4542addc-kube-api-access-b6ltm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s\" (UID: \"b3970d43-ba51-465c-8327-093a4542addc\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.615684 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.615571 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:11.752042 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.752014 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s"] Jul 24 15:46:11.753436 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:46:11.753409 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3970d43_ba51_465c_8327_093a4542addc.slice/crio-9fce1f76e72de6b5c40d943f72013d28c3ab70ba96da3b744c6580719e2e82a2 WatchSource:0}: Error finding container 9fce1f76e72de6b5c40d943f72013d28c3ab70ba96da3b744c6580719e2e82a2: Status 404 returned error can't find the container with id 9fce1f76e72de6b5c40d943f72013d28c3ab70ba96da3b744c6580719e2e82a2 Jul 24 15:46:11.754742 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:11.754724 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:46:12.284103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:12.284068 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" event={"ID":"b3970d43-ba51-465c-8327-093a4542addc","Type":"ContainerStarted","Data":"c5d07b03fc14d351243249db2ce39e56d4ff37c087fb0596aa4d295a4479cdf3"} Jul 24 15:46:12.284380 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:12.284121 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" event={"ID":"b3970d43-ba51-465c-8327-093a4542addc","Type":"ContainerStarted","Data":"9fce1f76e72de6b5c40d943f72013d28c3ab70ba96da3b744c6580719e2e82a2"} Jul 24 15:46:12.302012 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:12.301965 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" podStartSLOduration=1.11390608 podStartE2EDuration="1.301950356s" totalImagesPullingTime="188.044276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:12.29988093 +0000 UTC m=+613.532608611" watchObservedRunningTime="2026-07-24 15:46:12.301950356 +0000 UTC m=+613.534678064" Jul 24 15:46:17.994286 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:17.994246 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb"] Jul 24 15:46:18.000118 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.000101 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.003983 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.003962 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 24 15:46:18.010527 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.010507 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb"] Jul 24 15:46:18.044606 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.044578 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.044769 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.044631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqrs\" (UniqueName: \"kubernetes.io/projected/184a3192-c10e-40b2-b5a0-995440966ee0-kube-api-access-fjqrs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.044821 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.044777 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.044821 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.044803 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.044912 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.044824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.044912 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.044899 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184a3192-c10e-40b2-b5a0-995440966ee0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.146267 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.146233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.146267 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.146265 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.146490 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.146288 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.146490 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.146413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184a3192-c10e-40b2-b5a0-995440966ee0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.146490 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.146478 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.146604 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.146514 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqrs\" (UniqueName: \"kubernetes.io/projected/184a3192-c10e-40b2-b5a0-995440966ee0-kube-api-access-fjqrs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.146745 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.146726 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.146834 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.146771 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.146890 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.146845 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.148706 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.148677 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/184a3192-c10e-40b2-b5a0-995440966ee0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.149038 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.149021 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184a3192-c10e-40b2-b5a0-995440966ee0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.154725 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.154683 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqrs\" (UniqueName: \"kubernetes.io/projected/184a3192-c10e-40b2-b5a0-995440966ee0-kube-api-access-fjqrs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ndzsb\" (UID: \"184a3192-c10e-40b2-b5a0-995440966ee0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.312491 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.312408 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:18.453120 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:18.453088 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb"] Jul 24 15:46:18.455103 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:46:18.455079 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184a3192_c10e_40b2_b5a0_995440966ee0.slice/crio-02539656e8b6bc55cec7f67f26855d1254cdbfbe5d17a7082af317c30da67660 WatchSource:0}: Error finding container 02539656e8b6bc55cec7f67f26855d1254cdbfbe5d17a7082af317c30da67660: Status 404 returned error can't find the container with id 02539656e8b6bc55cec7f67f26855d1254cdbfbe5d17a7082af317c30da67660 Jul 24 15:46:19.323121 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:19.323043 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" event={"ID":"184a3192-c10e-40b2-b5a0-995440966ee0","Type":"ContainerStarted","Data":"122c08cc2e341bdfc52681b2a7fcf9a78065104ed4aeba1120bb8467398d9d35"} Jul 24 15:46:19.323121 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:19.323091 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" event={"ID":"184a3192-c10e-40b2-b5a0-995440966ee0","Type":"ContainerStarted","Data":"02539656e8b6bc55cec7f67f26855d1254cdbfbe5d17a7082af317c30da67660"} Jul 24 15:46:19.340945 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:19.340892 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" podStartSLOduration=2.114712292 podStartE2EDuration="2.340876747s" totalImagesPullingTime="226.164455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:19.33952463 +0000 UTC m=+620.572252312" watchObservedRunningTime="2026-07-24 15:46:19.340876747 +0000 UTC m=+620.573604433" Jul 24 15:46:21.616546 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:21.616489 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:21.617032 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:21.616571 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:21.629036 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:21.629013 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:22.350789 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:22.350764 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s" Jul 24 15:46:28.313295 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:28.313236 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:28.313760 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:28.313332 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:28.325778 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:28.325754 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:28.383907 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:28.383881 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ndzsb" Jul 24 15:46:31.604999 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.604965 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4"] Jul 24 15:46:31.608359 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.608338 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.610765 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.610740 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 24 15:46:31.618017 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.617993 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4"] Jul 24 15:46:31.660851 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.660814 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.660851 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.660849 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.661068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.660943 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.661068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.660981 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e429bb-6496-40e2-9896-16c8da5003b3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.661068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.660998 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.661068 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.661014 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kjqr\" (UniqueName: \"kubernetes.io/projected/d6e429bb-6496-40e2-9896-16c8da5003b3-kube-api-access-2kjqr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.762424 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.762385 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e429bb-6496-40e2-9896-16c8da5003b3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.762424 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.762420 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.762686 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.762439 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjqr\" (UniqueName: \"kubernetes.io/projected/d6e429bb-6496-40e2-9896-16c8da5003b3-kube-api-access-2kjqr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.762686 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.762474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.762686 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.762497 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.762686 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.762582 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.763007 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.762922 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.763007 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.762937 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.763090 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.763006 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.765010 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.764983 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e429bb-6496-40e2-9896-16c8da5003b3-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.765121 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.765106 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e429bb-6496-40e2-9896-16c8da5003b3-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.771103 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.771076 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjqr\" (UniqueName: \"kubernetes.io/projected/d6e429bb-6496-40e2-9896-16c8da5003b3-kube-api-access-2kjqr\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4\" (UID: \"d6e429bb-6496-40e2-9896-16c8da5003b3\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:31.919402 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:31.919316 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:32.058266 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:32.058240 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4"] Jul 24 15:46:32.059946 ip-10-0-132-156 kubenswrapper[2571]: W0724 15:46:32.059919 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6e429bb_6496_40e2_9896_16c8da5003b3.slice/crio-98a59fda11cb727640ca1890e31a56d5af06979c11006faf7bf329c29c11b2f4 WatchSource:0}: Error finding container 98a59fda11cb727640ca1890e31a56d5af06979c11006faf7bf329c29c11b2f4: Status 404 returned error can't find the container with id 98a59fda11cb727640ca1890e31a56d5af06979c11006faf7bf329c29c11b2f4 Jul 24 15:46:32.396898 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:32.396861 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" event={"ID":"d6e429bb-6496-40e2-9896-16c8da5003b3","Type":"ContainerStarted","Data":"9df233b0de85203ca68a8eff8a7274a65bcc0fdcac29c094c699cb75ac1b78d9"} Jul 24 15:46:32.397063 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:32.396915 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" event={"ID":"d6e429bb-6496-40e2-9896-16c8da5003b3","Type":"ContainerStarted","Data":"98a59fda11cb727640ca1890e31a56d5af06979c11006faf7bf329c29c11b2f4"} Jul 24 15:46:32.420282 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:32.420220 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" podStartSLOduration=1.202219781 podStartE2EDuration="1.420201874s" totalImagesPullingTime="217.982093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:32.418458131 +0000 UTC m=+633.651185836" watchObservedRunningTime="2026-07-24 15:46:32.420201874 +0000 UTC m=+633.652929610" Jul 24 15:46:41.919479 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:41.919435 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:41.919479 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:41.919492 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:41.932435 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:41.932409 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:46:42.466072 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:46:42.466038 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4" Jul 24 15:48:14.284922 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.284879 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67bb47cc5d-c6fhr"] Jul 24 15:48:14.285401 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.285138 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" podUID="fc3cfc5b-a93e-4384-8a93-6e351aa0f886" containerName="manager" containerID="cri-o://6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6" gracePeriod=30 Jul 24 15:48:14.545964 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.545907 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:48:14.682169 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.682134 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-webhook-cert\") pod \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\" (UID: \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\") " Jul 24 15:48:14.682169 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.682177 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nppfm\" (UniqueName: \"kubernetes.io/projected/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-kube-api-access-nppfm\") pod \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\" (UID: \"fc3cfc5b-a93e-4384-8a93-6e351aa0f886\") " Jul 24 15:48:14.684534 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.684492 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fc3cfc5b-a93e-4384-8a93-6e351aa0f886" (UID: "fc3cfc5b-a93e-4384-8a93-6e351aa0f886"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:48:14.684648 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.684596 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-kube-api-access-nppfm" (OuterVolumeSpecName: "kube-api-access-nppfm") pod "fc3cfc5b-a93e-4384-8a93-6e351aa0f886" (UID: "fc3cfc5b-a93e-4384-8a93-6e351aa0f886"). InnerVolumeSpecName "kube-api-access-nppfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:14.782991 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.782958 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-webhook-cert\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:48:14.782991 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.782984 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nppfm\" (UniqueName: \"kubernetes.io/projected/fc3cfc5b-a93e-4384-8a93-6e351aa0f886-kube-api-access-nppfm\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 15:48:14.945619 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.945524 2571 generic.go:356] "Generic (PLEG): container finished" podID="fc3cfc5b-a93e-4384-8a93-6e351aa0f886" containerID="6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6" exitCode=0 Jul 24 15:48:14.945619 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.945572 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" Jul 24 15:48:14.945838 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.945613 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" event={"ID":"fc3cfc5b-a93e-4384-8a93-6e351aa0f886","Type":"ContainerDied","Data":"6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6"} Jul 24 15:48:14.945838 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.945666 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-67bb47cc5d-c6fhr" event={"ID":"fc3cfc5b-a93e-4384-8a93-6e351aa0f886","Type":"ContainerDied","Data":"8d591b30ed132a454652abb1d60f137f9ed2e97c7382973599d593b1406fdfc0"} Jul 24 15:48:14.945838 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.945688 2571 scope.go:117] "RemoveContainer" containerID="6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6" Jul 24 15:48:14.955493 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.955478 2571 scope.go:117] "RemoveContainer" containerID="6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6" Jul 24 15:48:14.955759 ip-10-0-132-156 kubenswrapper[2571]: E0724 15:48:14.955737 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6\": container with ID starting with 6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6 not found: ID does not exist" containerID="6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6" Jul 24 15:48:14.955812 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.955766 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6"} err="failed to get container status \"6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6\": rpc error: code = NotFound desc = could not find container \"6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6\": container with ID starting with 6da488b6b1604c6aa4189fc01e8520c5d8cd3db45b56c3e990ec6fe80e6aa5d6 not found: ID does not exist" Jul 24 15:48:14.968529 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.968494 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-67bb47cc5d-c6fhr"] Jul 24 15:48:14.970827 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:14.970810 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-67bb47cc5d-c6fhr"] Jul 24 15:48:15.252304 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:48:15.252266 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc3cfc5b-a93e-4384-8a93-6e351aa0f886" path="/var/lib/kubelet/pods/fc3cfc5b-a93e-4384-8a93-6e351aa0f886/volumes" Jul 24 15:50:59.204733 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:50:59.204637 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:50:59.204733 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:50:59.204638 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:50:59.210166 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:50:59.210148 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:50:59.210316 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:50:59.210301 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:55:59.239615 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:55:59.239589 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:55:59.240762 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:55:59.240740 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 15:55:59.245422 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:55:59.245397 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 15:55:59.246957 ip-10-0-132-156 kubenswrapper[2571]: I0724 15:55:59.246935 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:00:59.273993 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:00:59.273968 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:00:59.277510 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:00:59.277491 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:00:59.279256 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:00:59.279235 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:00:59.282990 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:00:59.282970 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:02:31.396606 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.396574 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j"] Jul 24 16:02:31.397220 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.397151 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc3cfc5b-a93e-4384-8a93-6e351aa0f886" containerName="manager" Jul 24 16:02:31.397220 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.397171 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc3cfc5b-a93e-4384-8a93-6e351aa0f886" containerName="manager" Jul 24 16:02:31.397349 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.397268 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc3cfc5b-a93e-4384-8a93-6e351aa0f886" containerName="manager" Jul 24 16:02:31.400382 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.400352 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:31.404194 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.404171 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5ce23d5e-serving-cert\"" Jul 24 16:02:31.412210 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.412147 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j"] Jul 24 16:02:31.534579 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.534549 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccbc7438-1eca-4066-87ad-c24fe7721dbe-maas-api-tls\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:31.534579 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.534590 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbc7438-1eca-4066-87ad-c24fe7721dbe-openshift-service-ca\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:31.534835 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.534666 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgxrl\" (UniqueName: \"kubernetes.io/projected/ccbc7438-1eca-4066-87ad-c24fe7721dbe-kube-api-access-rgxrl\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:31.635689 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.635652 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccbc7438-1eca-4066-87ad-c24fe7721dbe-maas-api-tls\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:31.635894 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.635724 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbc7438-1eca-4066-87ad-c24fe7721dbe-openshift-service-ca\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:31.635894 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.635780 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgxrl\" (UniqueName: \"kubernetes.io/projected/ccbc7438-1eca-4066-87ad-c24fe7721dbe-kube-api-access-rgxrl\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:31.635894 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:02:31.635817 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-serving-cert: secret "maas-api-e2e-ait-5ce23d5e-serving-cert" not found Jul 24 16:02:31.635894 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:02:31.635893 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccbc7438-1eca-4066-87ad-c24fe7721dbe-maas-api-tls podName:ccbc7438-1eca-4066-87ad-c24fe7721dbe nodeName:}" failed. No retries permitted until 2026-07-24 16:02:32.135875152 +0000 UTC m=+1593.368602810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ccbc7438-1eca-4066-87ad-c24fe7721dbe-maas-api-tls") pod "maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" (UID: "ccbc7438-1eca-4066-87ad-c24fe7721dbe") : secret "maas-api-e2e-ait-5ce23d5e-serving-cert" not found Jul 24 16:02:31.636371 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.636349 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbc7438-1eca-4066-87ad-c24fe7721dbe-openshift-service-ca\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:31.649018 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:31.648955 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgxrl\" (UniqueName: \"kubernetes.io/projected/ccbc7438-1eca-4066-87ad-c24fe7721dbe-kube-api-access-rgxrl\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:32.141011 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:32.140970 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccbc7438-1eca-4066-87ad-c24fe7721dbe-maas-api-tls\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:32.143753 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:32.143724 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccbc7438-1eca-4066-87ad-c24fe7721dbe-maas-api-tls\") pod \"maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:32.313477 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:32.313442 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:32.454139 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:32.454111 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j"] Jul 24 16:02:32.455912 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:02:32.455885 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbc7438_1eca_4066_87ad_c24fe7721dbe.slice/crio-7520b108a4117540b4fe765caa8c6d1bd034f9a270c7df3a6f50f91e459854f6 WatchSource:0}: Error finding container 7520b108a4117540b4fe765caa8c6d1bd034f9a270c7df3a6f50f91e459854f6: Status 404 returned error can't find the container with id 7520b108a4117540b4fe765caa8c6d1bd034f9a270c7df3a6f50f91e459854f6 Jul 24 16:02:32.457294 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:32.457277 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:02:32.486662 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:32.486628 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" event={"ID":"ccbc7438-1eca-4066-87ad-c24fe7721dbe","Type":"ContainerStarted","Data":"7520b108a4117540b4fe765caa8c6d1bd034f9a270c7df3a6f50f91e459854f6"} Jul 24 16:02:33.494328 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:33.494291 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" event={"ID":"ccbc7438-1eca-4066-87ad-c24fe7721dbe","Type":"ContainerStarted","Data":"423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a"} Jul 24 16:02:33.494757 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:33.494359 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:02:33.512135 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:33.512034 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" podStartSLOduration=1.940989864 podStartE2EDuration="2.512015322s" totalImagesPullingTime="571.025458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:02:33.511302226 +0000 UTC m=+1594.744029904" watchObservedRunningTime="2026-07-24 16:02:33.512015322 +0000 UTC m=+1594.744743000" Jul 24 16:02:39.505508 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:02:39.505479 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:03:07.318650 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.318559 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j"] Jul 24 16:03:07.319263 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.318793 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" podUID="ccbc7438-1eca-4066-87ad-c24fe7721dbe" containerName="maas-api" containerID="cri-o://423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a" gracePeriod=30 Jul 24 16:03:07.574417 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.574362 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:03:07.653804 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.653774 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccbc7438-1eca-4066-87ad-c24fe7721dbe-maas-api-tls\") pod \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " Jul 24 16:03:07.653978 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.653869 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbc7438-1eca-4066-87ad-c24fe7721dbe-openshift-service-ca\") pod \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " Jul 24 16:03:07.653978 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.653907 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgxrl\" (UniqueName: \"kubernetes.io/projected/ccbc7438-1eca-4066-87ad-c24fe7721dbe-kube-api-access-rgxrl\") pod \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\" (UID: \"ccbc7438-1eca-4066-87ad-c24fe7721dbe\") " Jul 24 16:03:07.654226 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.654201 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccbc7438-1eca-4066-87ad-c24fe7721dbe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ccbc7438-1eca-4066-87ad-c24fe7721dbe" (UID: "ccbc7438-1eca-4066-87ad-c24fe7721dbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:03:07.656187 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.656166 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccbc7438-1eca-4066-87ad-c24fe7721dbe-kube-api-access-rgxrl" (OuterVolumeSpecName: "kube-api-access-rgxrl") pod "ccbc7438-1eca-4066-87ad-c24fe7721dbe" (UID: "ccbc7438-1eca-4066-87ad-c24fe7721dbe"). InnerVolumeSpecName "kube-api-access-rgxrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:03:07.656297 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.656279 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccbc7438-1eca-4066-87ad-c24fe7721dbe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ccbc7438-1eca-4066-87ad-c24fe7721dbe" (UID: "ccbc7438-1eca-4066-87ad-c24fe7721dbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:03:07.675368 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.675343 2571 generic.go:356] "Generic (PLEG): container finished" podID="ccbc7438-1eca-4066-87ad-c24fe7721dbe" containerID="423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a" exitCode=0 Jul 24 16:03:07.675477 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.675393 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" event={"ID":"ccbc7438-1eca-4066-87ad-c24fe7721dbe","Type":"ContainerDied","Data":"423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a"} Jul 24 16:03:07.675477 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.675395 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" Jul 24 16:03:07.675477 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.675433 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j" event={"ID":"ccbc7438-1eca-4066-87ad-c24fe7721dbe","Type":"ContainerDied","Data":"7520b108a4117540b4fe765caa8c6d1bd034f9a270c7df3a6f50f91e459854f6"} Jul 24 16:03:07.675477 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.675454 2571 scope.go:117] "RemoveContainer" containerID="423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a" Jul 24 16:03:07.686047 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.686024 2571 scope.go:117] "RemoveContainer" containerID="423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a" Jul 24 16:03:07.686339 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:03:07.686317 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a\": container with ID starting with 423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a not found: ID does not exist" containerID="423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a" Jul 24 16:03:07.686420 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.686347 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a"} err="failed to get container status \"423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a\": rpc error: code = NotFound desc = could not find container \"423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a\": container with ID starting with 423ef703f4dfc1923148681004813a5cd8972f40c7fa0e356781748db9ac400a not found: ID does not exist" Jul 24 16:03:07.699993 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.699972 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j"] Jul 24 16:03:07.705247 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.705224 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5ce23d5e-599bdf76fb-rhv5j"] Jul 24 16:03:07.754601 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.754577 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgxrl\" (UniqueName: \"kubernetes.io/projected/ccbc7438-1eca-4066-87ad-c24fe7721dbe-kube-api-access-rgxrl\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:03:07.754601 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.754598 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ccbc7438-1eca-4066-87ad-c24fe7721dbe-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:03:07.754771 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:07.754608 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccbc7438-1eca-4066-87ad-c24fe7721dbe-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:03:09.250610 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:09.250576 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccbc7438-1eca-4066-87ad-c24fe7721dbe" path="/var/lib/kubelet/pods/ccbc7438-1eca-4066-87ad-c24fe7721dbe/volumes" Jul 24 16:03:15.603263 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.603230 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8"] Jul 24 16:03:15.603636 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.603593 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccbc7438-1eca-4066-87ad-c24fe7721dbe" containerName="maas-api" Jul 24 16:03:15.603636 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.603604 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccbc7438-1eca-4066-87ad-c24fe7721dbe" containerName="maas-api" Jul 24 16:03:15.603744 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.603666 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccbc7438-1eca-4066-87ad-c24fe7721dbe" containerName="maas-api" Jul 24 16:03:15.608100 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.608082 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.610745 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.610686 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-09aa8ba8-serving-cert\"" Jul 24 16:03:15.615988 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.615949 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8"] Jul 24 16:03:15.726689 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.726651 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fnxb\" (UniqueName: \"kubernetes.io/projected/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-kube-api-access-5fnxb\") pod \"maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.726902 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.726814 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-openshift-service-ca\") pod \"maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.726902 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.726868 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-maas-api-tls\") pod \"maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.827671 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.827634 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fnxb\" (UniqueName: \"kubernetes.io/projected/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-kube-api-access-5fnxb\") pod \"maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.827918 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.827757 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-openshift-service-ca\") pod \"maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.827918 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.827819 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-maas-api-tls\") pod \"maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.828425 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.828392 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-openshift-service-ca\") pod \"maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.830496 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.830475 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-maas-api-tls\") pod \"maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.836360 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.836336 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fnxb\" (UniqueName: \"kubernetes.io/projected/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-kube-api-access-5fnxb\") pod \"maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:15.920171 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:15.920080 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:16.062632 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:16.062607 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8"] Jul 24 16:03:16.064618 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:03:16.064594 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf99d50_5da3_431b_bde8_e8f8e4f426ce.slice/crio-28c04da36e39d026984d62810e162481ca5b89ba7add3d09d118c8f7760a1212 WatchSource:0}: Error finding container 28c04da36e39d026984d62810e162481ca5b89ba7add3d09d118c8f7760a1212: Status 404 returned error can't find the container with id 28c04da36e39d026984d62810e162481ca5b89ba7add3d09d118c8f7760a1212 Jul 24 16:03:16.722996 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:16.722957 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" event={"ID":"ecf99d50-5da3-431b-bde8-e8f8e4f426ce","Type":"ContainerStarted","Data":"28c04da36e39d026984d62810e162481ca5b89ba7add3d09d118c8f7760a1212"} Jul 24 16:03:17.729299 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:17.729261 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" event={"ID":"ecf99d50-5da3-431b-bde8-e8f8e4f426ce","Type":"ContainerStarted","Data":"1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b"} Jul 24 16:03:17.729723 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:17.729327 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:17.749034 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:17.748983 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" podStartSLOduration=1.922773232 podStartE2EDuration="2.748967823s" totalImagesPullingTime="826.194591ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:03:17.745809761 +0000 UTC m=+1638.978537440" watchObservedRunningTime="2026-07-24 16:03:17.748967823 +0000 UTC m=+1638.981695537" Jul 24 16:03:23.739538 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:23.739511 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:51.033371 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.033340 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8"] Jul 24 16:03:51.033877 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.033600 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" podUID="ecf99d50-5da3-431b-bde8-e8f8e4f426ce" containerName="maas-api" containerID="cri-o://1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b" gracePeriod=30 Jul 24 16:03:51.289898 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.289839 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:51.363833 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.363805 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fnxb\" (UniqueName: \"kubernetes.io/projected/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-kube-api-access-5fnxb\") pod \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " Jul 24 16:03:51.363999 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.363861 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-maas-api-tls\") pod \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " Jul 24 16:03:51.363999 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.363922 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-openshift-service-ca\") pod \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\" (UID: \"ecf99d50-5da3-431b-bde8-e8f8e4f426ce\") " Jul 24 16:03:51.364360 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.364328 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ecf99d50-5da3-431b-bde8-e8f8e4f426ce" (UID: "ecf99d50-5da3-431b-bde8-e8f8e4f426ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:03:51.366155 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.366132 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ecf99d50-5da3-431b-bde8-e8f8e4f426ce" (UID: "ecf99d50-5da3-431b-bde8-e8f8e4f426ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:03:51.366270 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.366226 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-kube-api-access-5fnxb" (OuterVolumeSpecName: "kube-api-access-5fnxb") pod "ecf99d50-5da3-431b-bde8-e8f8e4f426ce" (UID: "ecf99d50-5da3-431b-bde8-e8f8e4f426ce"). InnerVolumeSpecName "kube-api-access-5fnxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:03:51.465247 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.465213 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fnxb\" (UniqueName: \"kubernetes.io/projected/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-kube-api-access-5fnxb\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:03:51.465247 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.465238 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:03:51.465247 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.465248 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecf99d50-5da3-431b-bde8-e8f8e4f426ce-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:03:51.918197 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.918157 2571 generic.go:356] "Generic (PLEG): container finished" podID="ecf99d50-5da3-431b-bde8-e8f8e4f426ce" containerID="1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b" exitCode=0 Jul 24 16:03:51.918407 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.918206 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" Jul 24 16:03:51.918407 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.918253 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" event={"ID":"ecf99d50-5da3-431b-bde8-e8f8e4f426ce","Type":"ContainerDied","Data":"1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b"} Jul 24 16:03:51.918407 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.918316 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8" event={"ID":"ecf99d50-5da3-431b-bde8-e8f8e4f426ce","Type":"ContainerDied","Data":"28c04da36e39d026984d62810e162481ca5b89ba7add3d09d118c8f7760a1212"} Jul 24 16:03:51.918407 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.918345 2571 scope.go:117] "RemoveContainer" containerID="1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b" Jul 24 16:03:51.928297 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.928282 2571 scope.go:117] "RemoveContainer" containerID="1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b" Jul 24 16:03:51.928592 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:03:51.928571 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b\": container with ID starting with 1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b not found: ID does not exist" containerID="1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b" Jul 24 16:03:51.928749 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.928598 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b"} err="failed to get container status \"1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b\": rpc error: code = NotFound desc = could not find container \"1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b\": container with ID starting with 1217cbdbc0628947e87b41285ebdcc1d00e2b43fa39c27dcf6323338c9641b5b not found: ID does not exist" Jul 24 16:03:51.940057 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.940036 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8"] Jul 24 16:03:51.943439 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:51.943417 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09aa8ba8-779c67f747-74ld8"] Jul 24 16:03:53.251101 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:53.251069 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf99d50-5da3-431b-bde8-e8f8e4f426ce" path="/var/lib/kubelet/pods/ecf99d50-5da3-431b-bde8-e8f8e4f426ce/volumes" Jul 24 16:03:58.801439 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.801409 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48"] Jul 24 16:03:58.801847 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.801803 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf99d50-5da3-431b-bde8-e8f8e4f426ce" containerName="maas-api" Jul 24 16:03:58.801847 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.801816 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf99d50-5da3-431b-bde8-e8f8e4f426ce" containerName="maas-api" Jul 24 16:03:58.801940 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.801925 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf99d50-5da3-431b-bde8-e8f8e4f426ce" containerName="maas-api" Jul 24 16:03:58.804931 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.804912 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:58.807594 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.807570 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-510804df-serving-cert\"" Jul 24 16:03:58.817250 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.817226 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48"] Jul 24 16:03:58.933444 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.933405 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsms\" (UniqueName: \"kubernetes.io/projected/9f30e04e-590a-412b-b4ed-40d54560c108-kube-api-access-6xsms\") pod \"maas-api-e2e-ait-510804df-55998d86f8-qsf48\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:58.933444 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.933448 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f30e04e-590a-412b-b4ed-40d54560c108-maas-api-tls\") pod \"maas-api-e2e-ait-510804df-55998d86f8-qsf48\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:58.933674 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:58.933487 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f30e04e-590a-412b-b4ed-40d54560c108-openshift-service-ca\") pod \"maas-api-e2e-ait-510804df-55998d86f8-qsf48\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:59.034522 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.034484 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsms\" (UniqueName: \"kubernetes.io/projected/9f30e04e-590a-412b-b4ed-40d54560c108-kube-api-access-6xsms\") pod \"maas-api-e2e-ait-510804df-55998d86f8-qsf48\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:59.034727 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.034529 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f30e04e-590a-412b-b4ed-40d54560c108-maas-api-tls\") pod \"maas-api-e2e-ait-510804df-55998d86f8-qsf48\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:59.034727 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.034660 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f30e04e-590a-412b-b4ed-40d54560c108-openshift-service-ca\") pod \"maas-api-e2e-ait-510804df-55998d86f8-qsf48\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:59.035288 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.035270 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f30e04e-590a-412b-b4ed-40d54560c108-openshift-service-ca\") pod \"maas-api-e2e-ait-510804df-55998d86f8-qsf48\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:59.037265 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.037228 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f30e04e-590a-412b-b4ed-40d54560c108-maas-api-tls\") pod \"maas-api-e2e-ait-510804df-55998d86f8-qsf48\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:59.043400 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.043378 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsms\" (UniqueName: \"kubernetes.io/projected/9f30e04e-590a-412b-b4ed-40d54560c108-kube-api-access-6xsms\") pod \"maas-api-e2e-ait-510804df-55998d86f8-qsf48\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:59.116605 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.116513 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:59.255264 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.255241 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48"] Jul 24 16:03:59.257021 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:03:59.256985 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f30e04e_590a_412b_b4ed_40d54560c108.slice/crio-7caa649835cb89f25c45247414a75d14f1f799446708554b3964c505714618d6 WatchSource:0}: Error finding container 7caa649835cb89f25c45247414a75d14f1f799446708554b3964c505714618d6: Status 404 returned error can't find the container with id 7caa649835cb89f25c45247414a75d14f1f799446708554b3964c505714618d6 Jul 24 16:03:59.962271 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.962183 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" event={"ID":"9f30e04e-590a-412b-b4ed-40d54560c108","Type":"ContainerStarted","Data":"23b2044567a16b4d2bbc1e9cb7f8c50fbc60044397521b52a895166047948829"} Jul 24 16:03:59.962271 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.962236 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" event={"ID":"9f30e04e-590a-412b-b4ed-40d54560c108","Type":"ContainerStarted","Data":"7caa649835cb89f25c45247414a75d14f1f799446708554b3964c505714618d6"} Jul 24 16:03:59.962717 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.962297 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:03:59.992513 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:03:59.989195 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" podStartSLOduration=1.585611994 podStartE2EDuration="1.989174751s" totalImagesPullingTime="403.562757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:03:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:03:59.984990678 +0000 UTC m=+1681.217718356" watchObservedRunningTime="2026-07-24 16:03:59.989174751 +0000 UTC m=+1681.221902430" Jul 24 16:04:05.973439 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:05.973410 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:04:34.974753 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:34.974495 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48"] Jul 24 16:04:34.976039 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:34.975963 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" podUID="9f30e04e-590a-412b-b4ed-40d54560c108" containerName="maas-api" containerID="cri-o://23b2044567a16b4d2bbc1e9cb7f8c50fbc60044397521b52a895166047948829" gracePeriod=30 Jul 24 16:04:35.152500 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.152456 2571 generic.go:356] "Generic (PLEG): container finished" podID="9f30e04e-590a-412b-b4ed-40d54560c108" containerID="23b2044567a16b4d2bbc1e9cb7f8c50fbc60044397521b52a895166047948829" exitCode=0 Jul 24 16:04:35.152661 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.152529 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" event={"ID":"9f30e04e-590a-412b-b4ed-40d54560c108","Type":"ContainerDied","Data":"23b2044567a16b4d2bbc1e9cb7f8c50fbc60044397521b52a895166047948829"} Jul 24 16:04:35.238821 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.238798 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:04:35.271665 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.271634 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f30e04e-590a-412b-b4ed-40d54560c108-maas-api-tls\") pod \"9f30e04e-590a-412b-b4ed-40d54560c108\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " Jul 24 16:04:35.271877 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.271743 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xsms\" (UniqueName: \"kubernetes.io/projected/9f30e04e-590a-412b-b4ed-40d54560c108-kube-api-access-6xsms\") pod \"9f30e04e-590a-412b-b4ed-40d54560c108\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " Jul 24 16:04:35.271877 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.271769 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f30e04e-590a-412b-b4ed-40d54560c108-openshift-service-ca\") pod \"9f30e04e-590a-412b-b4ed-40d54560c108\" (UID: \"9f30e04e-590a-412b-b4ed-40d54560c108\") " Jul 24 16:04:35.272198 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.272176 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f30e04e-590a-412b-b4ed-40d54560c108-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f30e04e-590a-412b-b4ed-40d54560c108" (UID: "9f30e04e-590a-412b-b4ed-40d54560c108"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:04:35.273995 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.273973 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f30e04e-590a-412b-b4ed-40d54560c108-kube-api-access-6xsms" (OuterVolumeSpecName: "kube-api-access-6xsms") pod "9f30e04e-590a-412b-b4ed-40d54560c108" (UID: "9f30e04e-590a-412b-b4ed-40d54560c108"). InnerVolumeSpecName "kube-api-access-6xsms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:04:35.274108 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.273994 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f30e04e-590a-412b-b4ed-40d54560c108-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f30e04e-590a-412b-b4ed-40d54560c108" (UID: "9f30e04e-590a-412b-b4ed-40d54560c108"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:04:35.373351 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.373302 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f30e04e-590a-412b-b4ed-40d54560c108-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:04:35.373351 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.373343 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xsms\" (UniqueName: \"kubernetes.io/projected/9f30e04e-590a-412b-b4ed-40d54560c108-kube-api-access-6xsms\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:04:35.373351 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:35.373352 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f30e04e-590a-412b-b4ed-40d54560c108-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:04:36.159062 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:36.159032 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" Jul 24 16:04:36.159522 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:36.159071 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48" event={"ID":"9f30e04e-590a-412b-b4ed-40d54560c108","Type":"ContainerDied","Data":"7caa649835cb89f25c45247414a75d14f1f799446708554b3964c505714618d6"} Jul 24 16:04:36.159522 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:36.159135 2571 scope.go:117] "RemoveContainer" containerID="23b2044567a16b4d2bbc1e9cb7f8c50fbc60044397521b52a895166047948829" Jul 24 16:04:36.182048 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:36.182017 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48"] Jul 24 16:04:36.185876 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:36.185854 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-510804df-55998d86f8-qsf48"] Jul 24 16:04:37.251176 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:37.251140 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f30e04e-590a-412b-b4ed-40d54560c108" path="/var/lib/kubelet/pods/9f30e04e-590a-412b-b4ed-40d54560c108/volumes" Jul 24 16:04:43.965622 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:43.965589 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl"] Jul 24 16:04:43.966091 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:43.965974 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f30e04e-590a-412b-b4ed-40d54560c108" containerName="maas-api" Jul 24 16:04:43.966091 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:43.965986 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f30e04e-590a-412b-b4ed-40d54560c108" containerName="maas-api" Jul 24 16:04:43.966091 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:43.966052 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f30e04e-590a-412b-b4ed-40d54560c108" containerName="maas-api" Jul 24 16:04:43.970294 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:43.970279 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:43.972913 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:43.972890 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-977c6dda-serving-cert\"" Jul 24 16:04:43.981971 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:43.981952 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl"] Jul 24 16:04:44.052240 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.052210 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44087c00-8287-4f2d-8f1a-5f4c86b43c23-maas-api-tls\") pod \"maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.052412 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.052246 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44087c00-8287-4f2d-8f1a-5f4c86b43c23-openshift-service-ca\") pod \"maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.052412 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.052286 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bbvx\" (UniqueName: \"kubernetes.io/projected/44087c00-8287-4f2d-8f1a-5f4c86b43c23-kube-api-access-7bbvx\") pod \"maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.153403 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.153355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bbvx\" (UniqueName: \"kubernetes.io/projected/44087c00-8287-4f2d-8f1a-5f4c86b43c23-kube-api-access-7bbvx\") pod \"maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.153592 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.153510 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44087c00-8287-4f2d-8f1a-5f4c86b43c23-maas-api-tls\") pod \"maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.153592 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.153551 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44087c00-8287-4f2d-8f1a-5f4c86b43c23-openshift-service-ca\") pod \"maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.154396 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.154371 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44087c00-8287-4f2d-8f1a-5f4c86b43c23-openshift-service-ca\") pod \"maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.156453 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.156431 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44087c00-8287-4f2d-8f1a-5f4c86b43c23-maas-api-tls\") pod \"maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.162967 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.162945 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bbvx\" (UniqueName: \"kubernetes.io/projected/44087c00-8287-4f2d-8f1a-5f4c86b43c23-kube-api-access-7bbvx\") pod \"maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.280318 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.280288 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:44.420419 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:44.420393 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl"] Jul 24 16:04:44.421769 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:04:44.421743 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44087c00_8287_4f2d_8f1a_5f4c86b43c23.slice/crio-bc8163bef99ff0844adad702c8e20a0c1e409ec65a126f5332e98d389141d8c8 WatchSource:0}: Error finding container bc8163bef99ff0844adad702c8e20a0c1e409ec65a126f5332e98d389141d8c8: Status 404 returned error can't find the container with id bc8163bef99ff0844adad702c8e20a0c1e409ec65a126f5332e98d389141d8c8 Jul 24 16:04:45.208792 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:45.208688 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" event={"ID":"44087c00-8287-4f2d-8f1a-5f4c86b43c23","Type":"ContainerStarted","Data":"dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe"} Jul 24 16:04:45.208792 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:45.208761 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" event={"ID":"44087c00-8287-4f2d-8f1a-5f4c86b43c23","Type":"ContainerStarted","Data":"bc8163bef99ff0844adad702c8e20a0c1e409ec65a126f5332e98d389141d8c8"} Jul 24 16:04:45.209465 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:45.208894 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:04:45.227018 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:45.226972 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" podStartSLOduration=1.750965525 podStartE2EDuration="2.226956699s" totalImagesPullingTime="475.991174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:04:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:04:45.223666198 +0000 UTC m=+1726.456393875" watchObservedRunningTime="2026-07-24 16:04:45.226956699 +0000 UTC m=+1726.459684396" Jul 24 16:04:51.219052 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:04:51.219021 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:05:19.395042 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.394994 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl"] Jul 24 16:05:19.395649 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.395230 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" podUID="44087c00-8287-4f2d-8f1a-5f4c86b43c23" containerName="maas-api" containerID="cri-o://dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe" gracePeriod=30 Jul 24 16:05:19.646532 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.646475 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:05:19.673750 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.673687 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44087c00-8287-4f2d-8f1a-5f4c86b43c23-openshift-service-ca\") pod \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " Jul 24 16:05:19.673933 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.673775 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44087c00-8287-4f2d-8f1a-5f4c86b43c23-maas-api-tls\") pod \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " Jul 24 16:05:19.673933 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.673855 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bbvx\" (UniqueName: \"kubernetes.io/projected/44087c00-8287-4f2d-8f1a-5f4c86b43c23-kube-api-access-7bbvx\") pod \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\" (UID: \"44087c00-8287-4f2d-8f1a-5f4c86b43c23\") " Jul 24 16:05:19.674054 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.674003 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44087c00-8287-4f2d-8f1a-5f4c86b43c23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "44087c00-8287-4f2d-8f1a-5f4c86b43c23" (UID: "44087c00-8287-4f2d-8f1a-5f4c86b43c23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:05:19.674166 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.674141 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44087c00-8287-4f2d-8f1a-5f4c86b43c23-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:05:19.676348 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.676306 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44087c00-8287-4f2d-8f1a-5f4c86b43c23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "44087c00-8287-4f2d-8f1a-5f4c86b43c23" (UID: "44087c00-8287-4f2d-8f1a-5f4c86b43c23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:05:19.676526 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.676503 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44087c00-8287-4f2d-8f1a-5f4c86b43c23-kube-api-access-7bbvx" (OuterVolumeSpecName: "kube-api-access-7bbvx") pod "44087c00-8287-4f2d-8f1a-5f4c86b43c23" (UID: "44087c00-8287-4f2d-8f1a-5f4c86b43c23"). InnerVolumeSpecName "kube-api-access-7bbvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:05:19.774982 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.774953 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44087c00-8287-4f2d-8f1a-5f4c86b43c23-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:05:19.774982 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:19.774979 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bbvx\" (UniqueName: \"kubernetes.io/projected/44087c00-8287-4f2d-8f1a-5f4c86b43c23-kube-api-access-7bbvx\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:05:20.404206 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:20.404173 2571 generic.go:356] "Generic (PLEG): container finished" podID="44087c00-8287-4f2d-8f1a-5f4c86b43c23" containerID="dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe" exitCode=0 Jul 24 16:05:20.404668 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:20.404241 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" Jul 24 16:05:20.404668 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:20.404242 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" event={"ID":"44087c00-8287-4f2d-8f1a-5f4c86b43c23","Type":"ContainerDied","Data":"dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe"} Jul 24 16:05:20.404668 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:20.404296 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl" event={"ID":"44087c00-8287-4f2d-8f1a-5f4c86b43c23","Type":"ContainerDied","Data":"bc8163bef99ff0844adad702c8e20a0c1e409ec65a126f5332e98d389141d8c8"} Jul 24 16:05:20.404668 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:20.404317 2571 scope.go:117] "RemoveContainer" containerID="dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe" Jul 24 16:05:20.413491 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:20.413475 2571 scope.go:117] "RemoveContainer" containerID="dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe" Jul 24 16:05:20.413778 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:05:20.413759 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe\": container with ID starting with dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe not found: ID does not exist" containerID="dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe" Jul 24 16:05:20.413863 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:20.413786 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe"} err="failed to get container status \"dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe\": rpc error: code = NotFound desc = could not find container \"dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe\": container with ID starting with dd7ed8bb32d931f27286ca2e2cfcfd93dd4a79a5d1a68196b844aeecf60d09fe not found: ID does not exist" Jul 24 16:05:20.426004 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:20.425982 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl"] Jul 24 16:05:20.430030 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:20.430010 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-977c6dda-6d8659b7bd-mgpbl"] Jul 24 16:05:21.250543 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:21.250512 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44087c00-8287-4f2d-8f1a-5f4c86b43c23" path="/var/lib/kubelet/pods/44087c00-8287-4f2d-8f1a-5f4c86b43c23/volumes" Jul 24 16:05:59.314046 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:59.314018 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:05:59.318764 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:59.318742 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:05:59.319463 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:59.319440 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:05:59.324264 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:05:59.324249 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:07:05.396984 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.396951 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht"] Jul 24 16:07:05.397485 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.397308 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44087c00-8287-4f2d-8f1a-5f4c86b43c23" containerName="maas-api" Jul 24 16:07:05.397485 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.397318 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="44087c00-8287-4f2d-8f1a-5f4c86b43c23" containerName="maas-api" Jul 24 16:07:05.397485 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.397389 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="44087c00-8287-4f2d-8f1a-5f4c86b43c23" containerName="maas-api" Jul 24 16:07:05.400553 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.400534 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.401803 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.401779 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f28jq\" (UniqueName: \"kubernetes.io/projected/b32b00b4-c771-4a61-88ae-39fe8dbc961d-kube-api-access-f28jq\") pod \"maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.401905 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.401891 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32b00b4-c771-4a61-88ae-39fe8dbc961d-maas-api-tls\") pod \"maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.401968 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.401934 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32b00b4-c771-4a61-88ae-39fe8dbc961d-openshift-service-ca\") pod \"maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.404811 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.404791 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-850ff15f-serving-cert\"" Jul 24 16:07:05.414160 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.414134 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht"] Jul 24 16:07:05.502487 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.502448 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32b00b4-c771-4a61-88ae-39fe8dbc961d-maas-api-tls\") pod \"maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.502739 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.502508 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32b00b4-c771-4a61-88ae-39fe8dbc961d-openshift-service-ca\") pod \"maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.502739 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.502547 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f28jq\" (UniqueName: \"kubernetes.io/projected/b32b00b4-c771-4a61-88ae-39fe8dbc961d-kube-api-access-f28jq\") pod \"maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.503216 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.503191 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32b00b4-c771-4a61-88ae-39fe8dbc961d-openshift-service-ca\") pod \"maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.505185 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.505159 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32b00b4-c771-4a61-88ae-39fe8dbc961d-maas-api-tls\") pod \"maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.515394 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.515372 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f28jq\" (UniqueName: \"kubernetes.io/projected/b32b00b4-c771-4a61-88ae-39fe8dbc961d-kube-api-access-f28jq\") pod \"maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.710786 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.710674 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:05.854600 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.854577 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht"] Jul 24 16:07:05.856868 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:07:05.856839 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32b00b4_c771_4a61_88ae_39fe8dbc961d.slice/crio-df8cd11441a6e27637ca533056b05253dca5f16134b062ed6af71d5d1b63620e WatchSource:0}: Error finding container df8cd11441a6e27637ca533056b05253dca5f16134b062ed6af71d5d1b63620e: Status 404 returned error can't find the container with id df8cd11441a6e27637ca533056b05253dca5f16134b062ed6af71d5d1b63620e Jul 24 16:07:05.973092 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:05.973014 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" event={"ID":"b32b00b4-c771-4a61-88ae-39fe8dbc961d","Type":"ContainerStarted","Data":"df8cd11441a6e27637ca533056b05253dca5f16134b062ed6af71d5d1b63620e"} Jul 24 16:07:07.985303 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:07.985263 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" event={"ID":"b32b00b4-c771-4a61-88ae-39fe8dbc961d","Type":"ContainerStarted","Data":"bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9"} Jul 24 16:07:07.985734 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:07.985326 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:08.004779 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:08.004718 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" podStartSLOduration=1.50566398 podStartE2EDuration="3.004690091s" totalImagesPullingTime="1.499026111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:07:08.002533306 +0000 UTC m=+1869.235261033" watchObservedRunningTime="2026-07-24 16:07:08.004690091 +0000 UTC m=+1869.237417769" Jul 24 16:07:13.996060 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:13.996029 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:27.729473 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:27.729388 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht"] Jul 24 16:07:27.729886 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:27.729645 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" podUID="b32b00b4-c771-4a61-88ae-39fe8dbc961d" containerName="maas-api" containerID="cri-o://bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9" gracePeriod=30 Jul 24 16:07:27.988555 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:27.988535 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:28.097892 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.097726 2571 generic.go:356] "Generic (PLEG): container finished" podID="b32b00b4-c771-4a61-88ae-39fe8dbc961d" containerID="bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9" exitCode=0 Jul 24 16:07:28.098086 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.097903 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" event={"ID":"b32b00b4-c771-4a61-88ae-39fe8dbc961d","Type":"ContainerDied","Data":"bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9"} Jul 24 16:07:28.098086 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.097951 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" event={"ID":"b32b00b4-c771-4a61-88ae-39fe8dbc961d","Type":"ContainerDied","Data":"df8cd11441a6e27637ca533056b05253dca5f16134b062ed6af71d5d1b63620e"} Jul 24 16:07:28.098086 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.097967 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht" Jul 24 16:07:28.098086 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.097974 2571 scope.go:117] "RemoveContainer" containerID="bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9" Jul 24 16:07:28.108054 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.108036 2571 scope.go:117] "RemoveContainer" containerID="bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9" Jul 24 16:07:28.108328 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:07:28.108310 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9\": container with ID starting with bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9 not found: ID does not exist" containerID="bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9" Jul 24 16:07:28.108397 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.108339 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9"} err="failed to get container status \"bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9\": rpc error: code = NotFound desc = could not find container \"bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9\": container with ID starting with bbd8032c0197ecbdb22e0d233cbda1a6d58d900d618324260d4058e5937d2ab9 not found: ID does not exist" Jul 24 16:07:28.108397 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.108354 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f28jq\" (UniqueName: \"kubernetes.io/projected/b32b00b4-c771-4a61-88ae-39fe8dbc961d-kube-api-access-f28jq\") pod \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " Jul 24 16:07:28.108521 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.108428 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32b00b4-c771-4a61-88ae-39fe8dbc961d-maas-api-tls\") pod \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " Jul 24 16:07:28.108521 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.108455 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32b00b4-c771-4a61-88ae-39fe8dbc961d-openshift-service-ca\") pod \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\" (UID: \"b32b00b4-c771-4a61-88ae-39fe8dbc961d\") " Jul 24 16:07:28.108872 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.108844 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32b00b4-c771-4a61-88ae-39fe8dbc961d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b32b00b4-c771-4a61-88ae-39fe8dbc961d" (UID: "b32b00b4-c771-4a61-88ae-39fe8dbc961d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:07:28.110564 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.110538 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32b00b4-c771-4a61-88ae-39fe8dbc961d-kube-api-access-f28jq" (OuterVolumeSpecName: "kube-api-access-f28jq") pod "b32b00b4-c771-4a61-88ae-39fe8dbc961d" (UID: "b32b00b4-c771-4a61-88ae-39fe8dbc961d"). InnerVolumeSpecName "kube-api-access-f28jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:07:28.110667 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.110619 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32b00b4-c771-4a61-88ae-39fe8dbc961d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b32b00b4-c771-4a61-88ae-39fe8dbc961d" (UID: "b32b00b4-c771-4a61-88ae-39fe8dbc961d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:07:28.209530 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.209489 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32b00b4-c771-4a61-88ae-39fe8dbc961d-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:07:28.209530 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.209522 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f28jq\" (UniqueName: \"kubernetes.io/projected/b32b00b4-c771-4a61-88ae-39fe8dbc961d-kube-api-access-f28jq\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:07:28.209530 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.209536 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32b00b4-c771-4a61-88ae-39fe8dbc961d-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:07:28.423115 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.423079 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht"] Jul 24 16:07:28.429954 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:28.429924 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-850ff15f-759bbd8c5d-7ckht"] Jul 24 16:07:29.251755 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:07:29.251726 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b32b00b4-c771-4a61-88ae-39fe8dbc961d" path="/var/lib/kubelet/pods/b32b00b4-c771-4a61-88ae-39fe8dbc961d/volumes" Jul 24 16:08:02.770719 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.770668 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg"] Jul 24 16:08:02.771243 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.771082 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32b00b4-c771-4a61-88ae-39fe8dbc961d" containerName="maas-api" Jul 24 16:08:02.771243 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.771097 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32b00b4-c771-4a61-88ae-39fe8dbc961d" containerName="maas-api" Jul 24 16:08:02.771243 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.771159 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b32b00b4-c771-4a61-88ae-39fe8dbc961d" containerName="maas-api" Jul 24 16:08:02.774238 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.774221 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:02.779756 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.779735 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5b382fb6-serving-cert\"" Jul 24 16:08:02.809271 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.809242 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg"] Jul 24 16:08:02.824866 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.824835 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjf52\" (UniqueName: \"kubernetes.io/projected/9a851e26-9895-44fa-8305-9d9f225ae8b5-kube-api-access-zjf52\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:02.825020 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.824891 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a851e26-9895-44fa-8305-9d9f225ae8b5-maas-api-tls\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:02.825020 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.824970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a851e26-9895-44fa-8305-9d9f225ae8b5-openshift-service-ca\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:02.925911 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.925871 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a851e26-9895-44fa-8305-9d9f225ae8b5-maas-api-tls\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:02.926122 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.925926 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a851e26-9895-44fa-8305-9d9f225ae8b5-openshift-service-ca\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:02.926122 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.925974 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjf52\" (UniqueName: \"kubernetes.io/projected/9a851e26-9895-44fa-8305-9d9f225ae8b5-kube-api-access-zjf52\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:02.926122 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:08:02.926046 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-serving-cert: secret "maas-api-e2e-mt-5b382fb6-serving-cert" not found Jul 24 16:08:02.926339 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:08:02.926131 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a851e26-9895-44fa-8305-9d9f225ae8b5-maas-api-tls podName:9a851e26-9895-44fa-8305-9d9f225ae8b5 nodeName:}" failed. No retries permitted until 2026-07-24 16:08:03.426109562 +0000 UTC m=+1924.658837219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9a851e26-9895-44fa-8305-9d9f225ae8b5-maas-api-tls") pod "maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" (UID: "9a851e26-9895-44fa-8305-9d9f225ae8b5") : secret "maas-api-e2e-mt-5b382fb6-serving-cert" not found Jul 24 16:08:02.926603 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.926584 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a851e26-9895-44fa-8305-9d9f225ae8b5-openshift-service-ca\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:02.947728 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:02.947667 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjf52\" (UniqueName: \"kubernetes.io/projected/9a851e26-9895-44fa-8305-9d9f225ae8b5-kube-api-access-zjf52\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:03.431303 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:03.431267 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a851e26-9895-44fa-8305-9d9f225ae8b5-maas-api-tls\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:03.433746 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:03.433716 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a851e26-9895-44fa-8305-9d9f225ae8b5-maas-api-tls\") pod \"maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:03.684976 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:03.684875 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:03.840534 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:03.840498 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg"] Jul 24 16:08:03.844022 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:08:03.843994 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a851e26_9895_44fa_8305_9d9f225ae8b5.slice/crio-78dd7d8092cd1e50878c00f1dc7be3086a4a7674832d948572acb635e472dac8 WatchSource:0}: Error finding container 78dd7d8092cd1e50878c00f1dc7be3086a4a7674832d948572acb635e472dac8: Status 404 returned error can't find the container with id 78dd7d8092cd1e50878c00f1dc7be3086a4a7674832d948572acb635e472dac8 Jul 24 16:08:03.845340 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:03.845320 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:08:04.298319 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:04.298283 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" event={"ID":"9a851e26-9895-44fa-8305-9d9f225ae8b5","Type":"ContainerStarted","Data":"0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0"} Jul 24 16:08:04.298500 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:04.298346 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:04.298500 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:04.298370 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" event={"ID":"9a851e26-9895-44fa-8305-9d9f225ae8b5","Type":"ContainerStarted","Data":"78dd7d8092cd1e50878c00f1dc7be3086a4a7674832d948572acb635e472dac8"} Jul 24 16:08:04.317070 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:04.317007 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" podStartSLOduration=2.005849903 podStartE2EDuration="2.316984925s" totalImagesPullingTime="311.135022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:08:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:08:04.316402527 +0000 UTC m=+1925.549130196" watchObservedRunningTime="2026-07-24 16:08:04.316984925 +0000 UTC m=+1925.549712606" Jul 24 16:08:10.309996 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:10.309962 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:12.695126 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.695084 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv"] Jul 24 16:08:12.698680 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.698665 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.702415 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.702390 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5b382fb6\"/\"default-dockercfg-5fd5z\"" Jul 24 16:08:12.702552 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.702462 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5b382fb6\"/\"openshift-service-ca.crt\"" Jul 24 16:08:12.702552 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.702485 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-5b382fb6\"/\"e2e-lifecycle-model-5b382fb6-kserve-self-signed-certs\"" Jul 24 16:08:12.702552 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.702462 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-5b382fb6\"/\"kube-root-ca.crt\"" Jul 24 16:08:12.708722 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.708673 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv"] Jul 24 16:08:12.815559 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.815530 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-dshm\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.815735 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.815574 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fzjl\" (UniqueName: \"kubernetes.io/projected/4b9f9cfa-e411-434a-9367-be26fda327d5-kube-api-access-8fzjl\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.815735 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.815606 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9f9cfa-e411-434a-9367-be26fda327d5-tls-certs\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.815735 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.815634 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-tmp-dir\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.815849 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.815734 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-home\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.815849 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.815752 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-model-cache\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.917054 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.917016 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-dshm\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.917054 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.917061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fzjl\" (UniqueName: \"kubernetes.io/projected/4b9f9cfa-e411-434a-9367-be26fda327d5-kube-api-access-8fzjl\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.917324 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.917086 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9f9cfa-e411-434a-9367-be26fda327d5-tls-certs\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.917324 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.917106 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-tmp-dir\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.917324 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.917154 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-home\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.917324 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.917171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-model-cache\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.917551 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.917530 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-tmp-dir\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.917594 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.917569 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-home\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.917645 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.917604 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-model-cache\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.919575 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.919555 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-dshm\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.919660 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.919650 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9f9cfa-e411-434a-9367-be26fda327d5-tls-certs\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:12.925810 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:12.925793 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fzjl\" (UniqueName: \"kubernetes.io/projected/4b9f9cfa-e411-434a-9367-be26fda327d5-kube-api-access-8fzjl\") pod \"e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:13.009023 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:13.008992 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:13.165845 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:13.165820 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv"] Jul 24 16:08:13.168641 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:08:13.168607 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b9f9cfa_e411_434a_9367_be26fda327d5.slice/crio-3f3d4943e1443e964c5ded933fd5f8540a7407b52681875f0cde5b283e8292ed WatchSource:0}: Error finding container 3f3d4943e1443e964c5ded933fd5f8540a7407b52681875f0cde5b283e8292ed: Status 404 returned error can't find the container with id 3f3d4943e1443e964c5ded933fd5f8540a7407b52681875f0cde5b283e8292ed Jul 24 16:08:13.348238 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:13.348146 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" event={"ID":"4b9f9cfa-e411-434a-9367-be26fda327d5","Type":"ContainerStarted","Data":"03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33"} Jul 24 16:08:13.348238 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:13.348192 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" event={"ID":"4b9f9cfa-e411-434a-9367-be26fda327d5","Type":"ContainerStarted","Data":"3f3d4943e1443e964c5ded933fd5f8540a7407b52681875f0cde5b283e8292ed"} Jul 24 16:08:13.371950 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:13.371903 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" podStartSLOduration=1.371886785 podStartE2EDuration="1.371886785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:08:13.370301863 +0000 UTC m=+1934.603029561" watchObservedRunningTime="2026-07-24 16:08:13.371886785 +0000 UTC m=+1934.604614463" Jul 24 16:08:23.009762 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:23.009711 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:23.009762 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:23.009764 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:23.023083 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:23.023054 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:23.412348 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:23.412265 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:08:52.082088 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.082008 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg"] Jul 24 16:08:52.084682 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.082277 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" podUID="9a851e26-9895-44fa-8305-9d9f225ae8b5" containerName="maas-api" containerID="cri-o://0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0" gracePeriod=30 Jul 24 16:08:52.335142 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.335079 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:52.357629 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.357596 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a851e26-9895-44fa-8305-9d9f225ae8b5-openshift-service-ca\") pod \"9a851e26-9895-44fa-8305-9d9f225ae8b5\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " Jul 24 16:08:52.357825 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.357656 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjf52\" (UniqueName: \"kubernetes.io/projected/9a851e26-9895-44fa-8305-9d9f225ae8b5-kube-api-access-zjf52\") pod \"9a851e26-9895-44fa-8305-9d9f225ae8b5\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " Jul 24 16:08:52.357825 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.357745 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a851e26-9895-44fa-8305-9d9f225ae8b5-maas-api-tls\") pod \"9a851e26-9895-44fa-8305-9d9f225ae8b5\" (UID: \"9a851e26-9895-44fa-8305-9d9f225ae8b5\") " Jul 24 16:08:52.358026 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.358001 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a851e26-9895-44fa-8305-9d9f225ae8b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a851e26-9895-44fa-8305-9d9f225ae8b5" (UID: "9a851e26-9895-44fa-8305-9d9f225ae8b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:08:52.358161 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.358144 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a851e26-9895-44fa-8305-9d9f225ae8b5-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:08:52.359966 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.359944 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a851e26-9895-44fa-8305-9d9f225ae8b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a851e26-9895-44fa-8305-9d9f225ae8b5" (UID: "9a851e26-9895-44fa-8305-9d9f225ae8b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:08:52.360368 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.360354 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a851e26-9895-44fa-8305-9d9f225ae8b5-kube-api-access-zjf52" (OuterVolumeSpecName: "kube-api-access-zjf52") pod "9a851e26-9895-44fa-8305-9d9f225ae8b5" (UID: "9a851e26-9895-44fa-8305-9d9f225ae8b5"). InnerVolumeSpecName "kube-api-access-zjf52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:08:52.459606 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.459576 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a851e26-9895-44fa-8305-9d9f225ae8b5-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:08:52.459606 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.459603 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjf52\" (UniqueName: \"kubernetes.io/projected/9a851e26-9895-44fa-8305-9d9f225ae8b5-kube-api-access-zjf52\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:08:52.561356 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.561325 2571 generic.go:356] "Generic (PLEG): container finished" podID="9a851e26-9895-44fa-8305-9d9f225ae8b5" containerID="0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0" exitCode=0 Jul 24 16:08:52.561545 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.561377 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" Jul 24 16:08:52.561545 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.561408 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" event={"ID":"9a851e26-9895-44fa-8305-9d9f225ae8b5","Type":"ContainerDied","Data":"0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0"} Jul 24 16:08:52.561545 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.561454 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg" event={"ID":"9a851e26-9895-44fa-8305-9d9f225ae8b5","Type":"ContainerDied","Data":"78dd7d8092cd1e50878c00f1dc7be3086a4a7674832d948572acb635e472dac8"} Jul 24 16:08:52.561545 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.561476 2571 scope.go:117] "RemoveContainer" containerID="0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0" Jul 24 16:08:52.571630 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.571613 2571 scope.go:117] "RemoveContainer" containerID="0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0" Jul 24 16:08:52.571912 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:08:52.571892 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0\": container with ID starting with 0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0 not found: ID does not exist" containerID="0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0" Jul 24 16:08:52.571980 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.571922 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0"} err="failed to get container status \"0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0\": rpc error: code = NotFound desc = could not find container \"0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0\": container with ID starting with 0c97ec2abd33c7245cc2769f489732b3db492394a30aa95e2ece4313cd6d0cf0 not found: ID does not exist" Jul 24 16:08:52.592155 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.592080 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg"] Jul 24 16:08:52.597596 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:52.597566 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5b382fb6-5d9cd455b5-87vmg"] Jul 24 16:08:53.251173 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:08:53.251142 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a851e26-9895-44fa-8305-9d9f225ae8b5" path="/var/lib/kubelet/pods/9a851e26-9895-44fa-8305-9d9f225ae8b5/volumes" Jul 24 16:09:05.144218 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:05.144181 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv"] Jul 24 16:09:20.164686 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.164631 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" podUID="4b9f9cfa-e411-434a-9367-be26fda327d5" containerName="main" containerID="cri-o://03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33" gracePeriod=45 Jul 24 16:09:20.438082 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.438054 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:09:20.531982 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.531949 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-home\") pod \"4b9f9cfa-e411-434a-9367-be26fda327d5\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " Jul 24 16:09:20.532181 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532006 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-dshm\") pod \"4b9f9cfa-e411-434a-9367-be26fda327d5\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " Jul 24 16:09:20.532181 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532027 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9f9cfa-e411-434a-9367-be26fda327d5-tls-certs\") pod \"4b9f9cfa-e411-434a-9367-be26fda327d5\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " Jul 24 16:09:20.532181 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532055 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-tmp-dir\") pod \"4b9f9cfa-e411-434a-9367-be26fda327d5\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " Jul 24 16:09:20.532181 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532105 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-model-cache\") pod \"4b9f9cfa-e411-434a-9367-be26fda327d5\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " Jul 24 16:09:20.532407 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532272 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fzjl\" (UniqueName: \"kubernetes.io/projected/4b9f9cfa-e411-434a-9367-be26fda327d5-kube-api-access-8fzjl\") pod \"4b9f9cfa-e411-434a-9367-be26fda327d5\" (UID: \"4b9f9cfa-e411-434a-9367-be26fda327d5\") " Jul 24 16:09:20.532407 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532330 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-home" (OuterVolumeSpecName: "home") pod "4b9f9cfa-e411-434a-9367-be26fda327d5" (UID: "4b9f9cfa-e411-434a-9367-be26fda327d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:20.532407 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532398 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b9f9cfa-e411-434a-9367-be26fda327d5" (UID: "4b9f9cfa-e411-434a-9367-be26fda327d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:20.532533 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532400 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b9f9cfa-e411-434a-9367-be26fda327d5" (UID: "4b9f9cfa-e411-434a-9367-be26fda327d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:20.532651 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532631 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-home\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:09:20.532651 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532650 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-tmp-dir\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:09:20.532651 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.532659 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-model-cache\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:09:20.534483 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.534453 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-dshm" (OuterVolumeSpecName: "dshm") pod "4b9f9cfa-e411-434a-9367-be26fda327d5" (UID: "4b9f9cfa-e411-434a-9367-be26fda327d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:20.534561 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.534500 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9f9cfa-e411-434a-9367-be26fda327d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b9f9cfa-e411-434a-9367-be26fda327d5" (UID: "4b9f9cfa-e411-434a-9367-be26fda327d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:09:20.534561 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.534509 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9f9cfa-e411-434a-9367-be26fda327d5-kube-api-access-8fzjl" (OuterVolumeSpecName: "kube-api-access-8fzjl") pod "4b9f9cfa-e411-434a-9367-be26fda327d5" (UID: "4b9f9cfa-e411-434a-9367-be26fda327d5"). InnerVolumeSpecName "kube-api-access-8fzjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:09:20.633786 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.633749 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fzjl\" (UniqueName: \"kubernetes.io/projected/4b9f9cfa-e411-434a-9367-be26fda327d5-kube-api-access-8fzjl\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:09:20.633786 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.633778 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b9f9cfa-e411-434a-9367-be26fda327d5-dshm\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:09:20.633786 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.633789 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b9f9cfa-e411-434a-9367-be26fda327d5-tls-certs\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:09:20.709028 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.708936 2571 generic.go:356] "Generic (PLEG): container finished" podID="4b9f9cfa-e411-434a-9367-be26fda327d5" containerID="03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33" exitCode=0 Jul 24 16:09:20.709028 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.708982 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" Jul 24 16:09:20.709028 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.708996 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" event={"ID":"4b9f9cfa-e411-434a-9367-be26fda327d5","Type":"ContainerDied","Data":"03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33"} Jul 24 16:09:20.709299 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.709036 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv" event={"ID":"4b9f9cfa-e411-434a-9367-be26fda327d5","Type":"ContainerDied","Data":"3f3d4943e1443e964c5ded933fd5f8540a7407b52681875f0cde5b283e8292ed"} Jul 24 16:09:20.709299 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.709055 2571 scope.go:117] "RemoveContainer" containerID="03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33" Jul 24 16:09:20.718848 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.718829 2571 scope.go:117] "RemoveContainer" containerID="03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33" Jul 24 16:09:20.719146 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:09:20.719124 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33\": container with ID starting with 03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33 not found: ID does not exist" containerID="03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33" Jul 24 16:09:20.719197 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.719161 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33"} err="failed to get container status \"03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33\": rpc error: code = NotFound desc = could not find container \"03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33\": container with ID starting with 03e6d675d3bc24d2f6643034400b895e501dd57a2ce7ef2897b82a6fa428ec33 not found: ID does not exist" Jul 24 16:09:20.731480 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.731454 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv"] Jul 24 16:09:20.734777 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:20.734755 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-5b382fb6/e2e-lifecycle-model-5b382fb6-kserve-7f9696cd67-jtdxv"] Jul 24 16:09:21.251422 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:09:21.251383 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b9f9cfa-e411-434a-9367-be26fda327d5" path="/var/lib/kubelet/pods/4b9f9cfa-e411-434a-9367-be26fda327d5/volumes" Jul 24 16:10:59.350545 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:10:59.350471 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:10:59.355061 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:10:59.355039 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:10:59.355827 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:10:59.355807 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:10:59.360483 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:10:59.360465 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:11:02.599016 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.598981 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln"] Jul 24 16:11:02.599497 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.599345 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9f9cfa-e411-434a-9367-be26fda327d5" containerName="main" Jul 24 16:11:02.599497 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.599355 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9f9cfa-e411-434a-9367-be26fda327d5" containerName="main" Jul 24 16:11:02.599497 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.599365 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a851e26-9895-44fa-8305-9d9f225ae8b5" containerName="maas-api" Jul 24 16:11:02.599497 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.599371 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a851e26-9895-44fa-8305-9d9f225ae8b5" containerName="maas-api" Jul 24 16:11:02.599497 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.599439 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9f9cfa-e411-434a-9367-be26fda327d5" containerName="main" Jul 24 16:11:02.599497 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.599446 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a851e26-9895-44fa-8305-9d9f225ae8b5" containerName="maas-api" Jul 24 16:11:02.607733 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.607712 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:02.610475 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.610452 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e03b82-serving-cert\"" Jul 24 16:11:02.615364 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.615091 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln"] Jul 24 16:11:02.658806 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.658775 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:02.658997 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.658829 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-maas-api-tls\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:02.658997 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.658847 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bfwn\" (UniqueName: \"kubernetes.io/projected/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-kube-api-access-9bfwn\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:02.759887 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.759847 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-maas-api-tls\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:02.759887 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.759885 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bfwn\" (UniqueName: \"kubernetes.io/projected/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-kube-api-access-9bfwn\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:02.760173 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.759951 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:02.760173 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:11:02.760022 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-serving-cert: secret "maas-api-e2e-inf-a-e03b82-serving-cert" not found Jul 24 16:11:02.760173 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:11:02.760100 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-maas-api-tls podName:ec8d9f67-81d2-4fd8-b90d-509a75fa8d33 nodeName:}" failed. No retries permitted until 2026-07-24 16:11:03.260079789 +0000 UTC m=+2104.492807460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-maas-api-tls") pod "maas-api-e2e-inf-a-e03b82-59558896f-7tgln" (UID: "ec8d9f67-81d2-4fd8-b90d-509a75fa8d33") : secret "maas-api-e2e-inf-a-e03b82-serving-cert" not found Jul 24 16:11:02.760610 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.760592 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:02.769811 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:02.769789 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bfwn\" (UniqueName: \"kubernetes.io/projected/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-kube-api-access-9bfwn\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:03.265367 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:03.265332 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-maas-api-tls\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:03.268015 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:03.267985 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-maas-api-tls\") pod \"maas-api-e2e-inf-a-e03b82-59558896f-7tgln\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:03.520026 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:03.519932 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:03.662953 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:03.662918 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln"] Jul 24 16:11:03.664501 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:11:03.664471 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8d9f67_81d2_4fd8_b90d_509a75fa8d33.slice/crio-bf73717de83ad326ac6b6e5ca81d740b4e96d90676779bfeaa8a8df549cb7e71 WatchSource:0}: Error finding container bf73717de83ad326ac6b6e5ca81d740b4e96d90676779bfeaa8a8df549cb7e71: Status 404 returned error can't find the container with id bf73717de83ad326ac6b6e5ca81d740b4e96d90676779bfeaa8a8df549cb7e71 Jul 24 16:11:04.261673 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:04.261636 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" event={"ID":"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33","Type":"ContainerStarted","Data":"bf73717de83ad326ac6b6e5ca81d740b4e96d90676779bfeaa8a8df549cb7e71"} Jul 24 16:11:05.268021 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:05.267979 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" event={"ID":"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33","Type":"ContainerStarted","Data":"e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b"} Jul 24 16:11:05.268508 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:05.268050 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:05.287598 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:05.287554 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" podStartSLOduration=2.630673639 podStartE2EDuration="3.287538433s" totalImagesPullingTime="656.864794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:11:05.284350653 +0000 UTC m=+2106.517078351" watchObservedRunningTime="2026-07-24 16:11:05.287538433 +0000 UTC m=+2106.520266148" Jul 24 16:11:10.717764 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.717724 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz"] Jul 24 16:11:10.721557 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.721540 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:10.724621 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.724592 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-aedd03-serving-cert\"" Jul 24 16:11:10.731620 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.731595 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz"] Jul 24 16:11:10.836070 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.836031 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0486fc-95c8-4f52-8038-d971348b4d0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:10.836273 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.836173 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwclw\" (UniqueName: \"kubernetes.io/projected/7d0486fc-95c8-4f52-8038-d971348b4d0d-kube-api-access-wwclw\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:10.836273 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.836248 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0486fc-95c8-4f52-8038-d971348b4d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:10.937504 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.937461 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0486fc-95c8-4f52-8038-d971348b4d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:10.937504 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.937504 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0486fc-95c8-4f52-8038-d971348b4d0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:10.937807 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.937564 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwclw\" (UniqueName: \"kubernetes.io/projected/7d0486fc-95c8-4f52-8038-d971348b4d0d-kube-api-access-wwclw\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:10.937807 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:11:10.937616 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-serving-cert: secret "maas-api-e2e-inf-b-aedd03-serving-cert" not found Jul 24 16:11:10.937807 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:11:10.937731 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d0486fc-95c8-4f52-8038-d971348b4d0d-maas-api-tls podName:7d0486fc-95c8-4f52-8038-d971348b4d0d nodeName:}" failed. No retries permitted until 2026-07-24 16:11:11.437681963 +0000 UTC m=+2112.670409621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7d0486fc-95c8-4f52-8038-d971348b4d0d-maas-api-tls") pod "maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" (UID: "7d0486fc-95c8-4f52-8038-d971348b4d0d") : secret "maas-api-e2e-inf-b-aedd03-serving-cert" not found Jul 24 16:11:10.938288 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.938268 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0486fc-95c8-4f52-8038-d971348b4d0d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:10.951574 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:10.951546 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwclw\" (UniqueName: \"kubernetes.io/projected/7d0486fc-95c8-4f52-8038-d971348b4d0d-kube-api-access-wwclw\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:11.278666 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:11.278633 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:11:11.441966 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:11.441933 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0486fc-95c8-4f52-8038-d971348b4d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:11.444545 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:11.444520 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0486fc-95c8-4f52-8038-d971348b4d0d-maas-api-tls\") pod \"maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:11.633772 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:11.633663 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:11.779944 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:11.779901 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz"] Jul 24 16:11:11.786025 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:11:11.785992 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0486fc_95c8_4f52_8038_d971348b4d0d.slice/crio-8b3e850dfa28b3601dc17ac72bd34b9c2ecbbdaba7876000515388f8afb60ea9 WatchSource:0}: Error finding container 8b3e850dfa28b3601dc17ac72bd34b9c2ecbbdaba7876000515388f8afb60ea9: Status 404 returned error can't find the container with id 8b3e850dfa28b3601dc17ac72bd34b9c2ecbbdaba7876000515388f8afb60ea9 Jul 24 16:11:12.306281 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:12.306244 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" event={"ID":"7d0486fc-95c8-4f52-8038-d971348b4d0d","Type":"ContainerStarted","Data":"29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244"} Jul 24 16:11:12.306483 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:12.306293 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" event={"ID":"7d0486fc-95c8-4f52-8038-d971348b4d0d","Type":"ContainerStarted","Data":"8b3e850dfa28b3601dc17ac72bd34b9c2ecbbdaba7876000515388f8afb60ea9"} Jul 24 16:11:12.306483 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:12.306392 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:12.327801 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:12.327685 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" podStartSLOduration=2.034811205 podStartE2EDuration="2.327667809s" totalImagesPullingTime="292.856604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:11:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:11:12.325915209 +0000 UTC m=+2113.558642887" watchObservedRunningTime="2026-07-24 16:11:12.327667809 +0000 UTC m=+2113.560395486" Jul 24 16:11:18.320506 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:18.320469 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:11:55.327279 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.327202 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6"] Jul 24 16:11:55.330883 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.330864 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.335286 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.335263 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-aedd03\"/\"kube-root-ca.crt\"" Jul 24 16:11:55.336741 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.336716 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-aedd03\"/\"default-dockercfg-r8hbp\"" Jul 24 16:11:55.336866 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.336716 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-aedd03\"/\"openshift-service-ca.crt\"" Jul 24 16:11:55.336866 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.336719 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-aedd03\"/\"test-model-aedd03-kserve-self-signed-certs\"" Jul 24 16:11:55.354662 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.354622 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6"] Jul 24 16:11:55.458299 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.458268 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tls-certs\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.458476 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.458332 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9dmb\" (UniqueName: \"kubernetes.io/projected/a52d4ac4-b14d-4188-aef1-2eb9274f362e-kube-api-access-k9dmb\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.458476 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.458438 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-model-cache\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.458558 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.458475 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-home\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.458558 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.458497 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-dshm\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.458558 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.458528 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tmp-dir\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.559825 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.559793 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tls-certs\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.560012 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.559857 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9dmb\" (UniqueName: \"kubernetes.io/projected/a52d4ac4-b14d-4188-aef1-2eb9274f362e-kube-api-access-k9dmb\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.560012 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.559898 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-model-cache\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.560012 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.559921 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-home\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.560012 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.559940 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-dshm\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.560223 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.560075 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tmp-dir\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.560401 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.560382 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-model-cache\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.560506 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.560418 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-home\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.560506 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.560476 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tmp-dir\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.562451 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.562426 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-dshm\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.562555 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.562514 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tls-certs\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.568964 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.568947 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9dmb\" (UniqueName: \"kubernetes.io/projected/a52d4ac4-b14d-4188-aef1-2eb9274f362e-kube-api-access-k9dmb\") pod \"test-model-aedd03-kserve-69b65595bc-bv5l6\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.640396 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.640306 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:11:55.790202 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:55.790171 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6"] Jul 24 16:11:55.791392 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:11:55.791364 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda52d4ac4_b14d_4188_aef1_2eb9274f362e.slice/crio-379e77283288245ece53b2e1dd71831ba962afb25fc39c62f490ee837dc92341 WatchSource:0}: Error finding container 379e77283288245ece53b2e1dd71831ba962afb25fc39c62f490ee837dc92341: Status 404 returned error can't find the container with id 379e77283288245ece53b2e1dd71831ba962afb25fc39c62f490ee837dc92341 Jul 24 16:11:56.551686 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:56.551645 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" event={"ID":"a52d4ac4-b14d-4188-aef1-2eb9274f362e","Type":"ContainerStarted","Data":"88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151"} Jul 24 16:11:56.551686 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:56.551708 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" event={"ID":"a52d4ac4-b14d-4188-aef1-2eb9274f362e","Type":"ContainerStarted","Data":"379e77283288245ece53b2e1dd71831ba962afb25fc39c62f490ee837dc92341"} Jul 24 16:11:56.576540 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:11:56.576495 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" podStartSLOduration=1.576470446 podStartE2EDuration="1.576470446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:11:56.576108547 +0000 UTC m=+2157.808836237" watchObservedRunningTime="2026-07-24 16:11:56.576470446 +0000 UTC m=+2157.809198166" Jul 24 16:12:05.641021 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:05.640981 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:12:05.641629 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:05.641043 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:12:05.653615 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:05.653587 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:12:06.623195 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:06.623163 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:12:41.294335 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:41.294300 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6"] Jul 24 16:12:52.060104 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.060073 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln"] Jul 24 16:12:52.060594 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.060361 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" podUID="ec8d9f67-81d2-4fd8-b90d-509a75fa8d33" containerName="maas-api" containerID="cri-o://e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b" gracePeriod=30 Jul 24 16:12:52.519104 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.519075 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:12:52.694899 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.694813 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-openshift-service-ca\") pod \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " Jul 24 16:12:52.694899 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.694899 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-maas-api-tls\") pod \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " Jul 24 16:12:52.695129 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.695071 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bfwn\" (UniqueName: \"kubernetes.io/projected/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-kube-api-access-9bfwn\") pod \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\" (UID: \"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33\") " Jul 24 16:12:52.695257 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.695234 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec8d9f67-81d2-4fd8-b90d-509a75fa8d33" (UID: "ec8d9f67-81d2-4fd8-b90d-509a75fa8d33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:12:52.695429 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.695414 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:12:52.697158 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.697136 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec8d9f67-81d2-4fd8-b90d-509a75fa8d33" (UID: "ec8d9f67-81d2-4fd8-b90d-509a75fa8d33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:12:52.697240 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.697217 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-kube-api-access-9bfwn" (OuterVolumeSpecName: "kube-api-access-9bfwn") pod "ec8d9f67-81d2-4fd8-b90d-509a75fa8d33" (UID: "ec8d9f67-81d2-4fd8-b90d-509a75fa8d33"). InnerVolumeSpecName "kube-api-access-9bfwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:12:52.796595 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.796560 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:12:52.796595 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.796590 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bfwn\" (UniqueName: \"kubernetes.io/projected/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33-kube-api-access-9bfwn\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:12:52.864417 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.864381 2571 generic.go:356] "Generic (PLEG): container finished" podID="ec8d9f67-81d2-4fd8-b90d-509a75fa8d33" containerID="e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b" exitCode=0 Jul 24 16:12:52.864624 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.864431 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" Jul 24 16:12:52.864624 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.864468 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" event={"ID":"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33","Type":"ContainerDied","Data":"e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b"} Jul 24 16:12:52.864624 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.864530 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln" event={"ID":"ec8d9f67-81d2-4fd8-b90d-509a75fa8d33","Type":"ContainerDied","Data":"bf73717de83ad326ac6b6e5ca81d740b4e96d90676779bfeaa8a8df549cb7e71"} Jul 24 16:12:52.864624 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.864557 2571 scope.go:117] "RemoveContainer" containerID="e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b" Jul 24 16:12:52.874533 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.874512 2571 scope.go:117] "RemoveContainer" containerID="e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b" Jul 24 16:12:52.874870 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:12:52.874848 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b\": container with ID starting with e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b not found: ID does not exist" containerID="e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b" Jul 24 16:12:52.874953 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.874878 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b"} err="failed to get container status \"e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b\": rpc error: code = NotFound desc = could not find container \"e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b\": container with ID starting with e964b909cb24437c80a99e1e8143bf13c9d8b5f5b3c138dedc8f02598bcd520b not found: ID does not exist" Jul 24 16:12:52.886246 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.886213 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln"] Jul 24 16:12:52.890448 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:52.890426 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e03b82-59558896f-7tgln"] Jul 24 16:12:53.250985 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:53.250952 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec8d9f67-81d2-4fd8-b90d-509a75fa8d33" path="/var/lib/kubelet/pods/ec8d9f67-81d2-4fd8-b90d-509a75fa8d33/volumes" Jul 24 16:12:56.313888 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.313834 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" podUID="a52d4ac4-b14d-4188-aef1-2eb9274f362e" containerName="main" containerID="cri-o://88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151" gracePeriod=45 Jul 24 16:12:56.575235 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.575177 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:12:56.732887 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.732849 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-model-cache\") pod \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " Jul 24 16:12:56.732887 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.732891 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tmp-dir\") pod \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " Jul 24 16:12:56.733141 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.732943 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tls-certs\") pod \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " Jul 24 16:12:56.733141 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.732963 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9dmb\" (UniqueName: \"kubernetes.io/projected/a52d4ac4-b14d-4188-aef1-2eb9274f362e-kube-api-access-k9dmb\") pod \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " Jul 24 16:12:56.733141 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.733002 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-dshm\") pod \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " Jul 24 16:12:56.733312 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.733160 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a52d4ac4-b14d-4188-aef1-2eb9274f362e" (UID: "a52d4ac4-b14d-4188-aef1-2eb9274f362e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:12:56.733312 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.733197 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-home\") pod \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\" (UID: \"a52d4ac4-b14d-4188-aef1-2eb9274f362e\") " Jul 24 16:12:56.733312 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.733201 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a52d4ac4-b14d-4188-aef1-2eb9274f362e" (UID: "a52d4ac4-b14d-4188-aef1-2eb9274f362e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:12:56.733480 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.733413 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-home" (OuterVolumeSpecName: "home") pod "a52d4ac4-b14d-4188-aef1-2eb9274f362e" (UID: "a52d4ac4-b14d-4188-aef1-2eb9274f362e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:12:56.733572 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.733554 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-home\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:12:56.733614 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.733578 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-model-cache\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:12:56.733614 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.733594 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tmp-dir\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:12:56.735372 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.735342 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-dshm" (OuterVolumeSpecName: "dshm") pod "a52d4ac4-b14d-4188-aef1-2eb9274f362e" (UID: "a52d4ac4-b14d-4188-aef1-2eb9274f362e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:12:56.735505 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.735427 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a52d4ac4-b14d-4188-aef1-2eb9274f362e" (UID: "a52d4ac4-b14d-4188-aef1-2eb9274f362e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:12:56.735505 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.735463 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a52d4ac4-b14d-4188-aef1-2eb9274f362e-kube-api-access-k9dmb" (OuterVolumeSpecName: "kube-api-access-k9dmb") pod "a52d4ac4-b14d-4188-aef1-2eb9274f362e" (UID: "a52d4ac4-b14d-4188-aef1-2eb9274f362e"). InnerVolumeSpecName "kube-api-access-k9dmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:12:56.835104 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.835015 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a52d4ac4-b14d-4188-aef1-2eb9274f362e-tls-certs\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:12:56.835104 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.835047 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9dmb\" (UniqueName: \"kubernetes.io/projected/a52d4ac4-b14d-4188-aef1-2eb9274f362e-kube-api-access-k9dmb\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:12:56.835104 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.835057 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a52d4ac4-b14d-4188-aef1-2eb9274f362e-dshm\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:12:56.887961 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.887927 2571 generic.go:356] "Generic (PLEG): container finished" podID="a52d4ac4-b14d-4188-aef1-2eb9274f362e" containerID="88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151" exitCode=0 Jul 24 16:12:56.888133 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.887968 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" Jul 24 16:12:56.888133 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.888018 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" event={"ID":"a52d4ac4-b14d-4188-aef1-2eb9274f362e","Type":"ContainerDied","Data":"88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151"} Jul 24 16:12:56.888133 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.888063 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6" event={"ID":"a52d4ac4-b14d-4188-aef1-2eb9274f362e","Type":"ContainerDied","Data":"379e77283288245ece53b2e1dd71831ba962afb25fc39c62f490ee837dc92341"} Jul 24 16:12:56.888133 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.888084 2571 scope.go:117] "RemoveContainer" containerID="88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151" Jul 24 16:12:56.897727 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.897689 2571 scope.go:117] "RemoveContainer" containerID="88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151" Jul 24 16:12:56.897965 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:12:56.897949 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151\": container with ID starting with 88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151 not found: ID does not exist" containerID="88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151" Jul 24 16:12:56.898015 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.897972 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151"} err="failed to get container status \"88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151\": rpc error: code = NotFound desc = could not find container \"88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151\": container with ID starting with 88de1b13cbaa1d58382a0d506aede233d0e228a0e4e228216cc4c5e0bc775151 not found: ID does not exist" Jul 24 16:12:56.914750 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.914718 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6"] Jul 24 16:12:56.917010 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:56.916984 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-aedd03/test-model-aedd03-kserve-69b65595bc-bv5l6"] Jul 24 16:12:57.251388 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:12:57.251345 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a52d4ac4-b14d-4188-aef1-2eb9274f362e" path="/var/lib/kubelet/pods/a52d4ac4-b14d-4188-aef1-2eb9274f362e/volumes" Jul 24 16:13:14.442121 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.442084 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz"] Jul 24 16:13:14.442568 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.442305 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" podUID="7d0486fc-95c8-4f52-8038-d971348b4d0d" containerName="maas-api" containerID="cri-o://29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244" gracePeriod=30 Jul 24 16:13:14.696282 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.696225 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:13:14.797493 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.797458 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwclw\" (UniqueName: \"kubernetes.io/projected/7d0486fc-95c8-4f52-8038-d971348b4d0d-kube-api-access-wwclw\") pod \"7d0486fc-95c8-4f52-8038-d971348b4d0d\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " Jul 24 16:13:14.797739 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.797509 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0486fc-95c8-4f52-8038-d971348b4d0d-openshift-service-ca\") pod \"7d0486fc-95c8-4f52-8038-d971348b4d0d\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " Jul 24 16:13:14.797739 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.797558 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0486fc-95c8-4f52-8038-d971348b4d0d-maas-api-tls\") pod \"7d0486fc-95c8-4f52-8038-d971348b4d0d\" (UID: \"7d0486fc-95c8-4f52-8038-d971348b4d0d\") " Jul 24 16:13:14.797968 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.797943 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0486fc-95c8-4f52-8038-d971348b4d0d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7d0486fc-95c8-4f52-8038-d971348b4d0d" (UID: "7d0486fc-95c8-4f52-8038-d971348b4d0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:13:14.799860 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.799835 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0486fc-95c8-4f52-8038-d971348b4d0d-kube-api-access-wwclw" (OuterVolumeSpecName: "kube-api-access-wwclw") pod "7d0486fc-95c8-4f52-8038-d971348b4d0d" (UID: "7d0486fc-95c8-4f52-8038-d971348b4d0d"). InnerVolumeSpecName "kube-api-access-wwclw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:13:14.799971 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.799930 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0486fc-95c8-4f52-8038-d971348b4d0d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7d0486fc-95c8-4f52-8038-d971348b4d0d" (UID: "7d0486fc-95c8-4f52-8038-d971348b4d0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:13:14.898401 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.898363 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7d0486fc-95c8-4f52-8038-d971348b4d0d-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:13:14.898401 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.898396 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwclw\" (UniqueName: \"kubernetes.io/projected/7d0486fc-95c8-4f52-8038-d971348b4d0d-kube-api-access-wwclw\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:13:14.898401 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.898406 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d0486fc-95c8-4f52-8038-d971348b4d0d-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:13:14.983810 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.983719 2571 generic.go:356] "Generic (PLEG): container finished" podID="7d0486fc-95c8-4f52-8038-d971348b4d0d" containerID="29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244" exitCode=0 Jul 24 16:13:14.983810 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.983757 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" Jul 24 16:13:14.983810 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.983802 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" event={"ID":"7d0486fc-95c8-4f52-8038-d971348b4d0d","Type":"ContainerDied","Data":"29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244"} Jul 24 16:13:14.984032 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.983857 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz" event={"ID":"7d0486fc-95c8-4f52-8038-d971348b4d0d","Type":"ContainerDied","Data":"8b3e850dfa28b3601dc17ac72bd34b9c2ecbbdaba7876000515388f8afb60ea9"} Jul 24 16:13:14.984032 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.983884 2571 scope.go:117] "RemoveContainer" containerID="29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244" Jul 24 16:13:14.994190 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.994171 2571 scope.go:117] "RemoveContainer" containerID="29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244" Jul 24 16:13:14.994464 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:13:14.994446 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244\": container with ID starting with 29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244 not found: ID does not exist" containerID="29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244" Jul 24 16:13:14.994512 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:14.994471 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244"} err="failed to get container status \"29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244\": rpc error: code = NotFound desc = could not find container \"29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244\": container with ID starting with 29a25a12186e4820eaf8736f3938a799dac00a6ea6f73de21fa9dea2425f2244 not found: ID does not exist" Jul 24 16:13:15.013451 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:15.013421 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz"] Jul 24 16:13:15.018326 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:15.018302 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-aedd03-f7fdcc9bc-5tckz"] Jul 24 16:13:15.251152 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:15.251124 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d0486fc-95c8-4f52-8038-d971348b4d0d" path="/var/lib/kubelet/pods/7d0486fc-95c8-4f52-8038-d971348b4d0d/volumes" Jul 24 16:13:31.497394 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.497314 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc"] Jul 24 16:13:31.497886 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.497870 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a52d4ac4-b14d-4188-aef1-2eb9274f362e" containerName="main" Jul 24 16:13:31.497936 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.497890 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="a52d4ac4-b14d-4188-aef1-2eb9274f362e" containerName="main" Jul 24 16:13:31.497936 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.497928 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0486fc-95c8-4f52-8038-d971348b4d0d" containerName="maas-api" Jul 24 16:13:31.498001 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.497937 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0486fc-95c8-4f52-8038-d971348b4d0d" containerName="maas-api" Jul 24 16:13:31.498001 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.497954 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8d9f67-81d2-4fd8-b90d-509a75fa8d33" containerName="maas-api" Jul 24 16:13:31.498001 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.497963 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8d9f67-81d2-4fd8-b90d-509a75fa8d33" containerName="maas-api" Jul 24 16:13:31.498089 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.498051 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8d9f67-81d2-4fd8-b90d-509a75fa8d33" containerName="maas-api" Jul 24 16:13:31.498089 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.498066 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="a52d4ac4-b14d-4188-aef1-2eb9274f362e" containerName="main" Jul 24 16:13:31.498089 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.498076 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d0486fc-95c8-4f52-8038-d971348b4d0d" containerName="maas-api" Jul 24 16:13:31.502674 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.502650 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.505450 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.505427 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-57ceec-serving-cert\"" Jul 24 16:13:31.515620 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.515597 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc"] Jul 24 16:13:31.647893 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.647861 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-openshift-service-ca\") pod \"maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.648083 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.647922 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-maas-api-tls\") pod \"maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.648083 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.648007 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl7zs\" (UniqueName: \"kubernetes.io/projected/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-kube-api-access-fl7zs\") pod \"maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.749019 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.748917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl7zs\" (UniqueName: \"kubernetes.io/projected/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-kube-api-access-fl7zs\") pod \"maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.749019 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.749009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-openshift-service-ca\") pod \"maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.749262 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.749066 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-maas-api-tls\") pod \"maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.749649 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.749627 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-openshift-service-ca\") pod \"maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.751684 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.751660 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-maas-api-tls\") pod \"maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.781042 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.781011 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl7zs\" (UniqueName: \"kubernetes.io/projected/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-kube-api-access-fl7zs\") pod \"maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.813909 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.813877 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:31.968350 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.968180 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc"] Jul 24 16:13:31.970690 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:13:31.970664 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d33cb02_cde0_42df_93ec_9a3b31eb8fa3.slice/crio-43432c202645d93f73c2f1da49b771865d92e41793973f6a4ede4e0183724bab WatchSource:0}: Error finding container 43432c202645d93f73c2f1da49b771865d92e41793973f6a4ede4e0183724bab: Status 404 returned error can't find the container with id 43432c202645d93f73c2f1da49b771865d92e41793973f6a4ede4e0183724bab Jul 24 16:13:31.972128 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:31.972111 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:13:32.073045 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:32.072956 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" event={"ID":"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3","Type":"ContainerStarted","Data":"43432c202645d93f73c2f1da49b771865d92e41793973f6a4ede4e0183724bab"} Jul 24 16:13:33.079820 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:33.079784 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" event={"ID":"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3","Type":"ContainerStarted","Data":"9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7"} Jul 24 16:13:33.080303 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:33.079870 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:33.112346 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:33.112298 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" podStartSLOduration=1.585662611 podStartE2EDuration="2.112282531s" totalImagesPullingTime="526.61992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:13:33.11011906 +0000 UTC m=+2254.342846739" watchObservedRunningTime="2026-07-24 16:13:33.112282531 +0000 UTC m=+2254.345010238" Jul 24 16:13:39.091673 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:39.091646 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:13:40.618158 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.618126 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88"] Jul 24 16:13:40.621818 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.621787 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.627494 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.627469 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-afdeb1-serving-cert\"" Jul 24 16:13:40.649507 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.649484 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88"] Jul 24 16:13:40.729638 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.729604 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a957598-1727-4fec-b5ac-29a407ec8c11-maas-api-tls\") pod \"maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.729848 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.729652 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqzk\" (UniqueName: \"kubernetes.io/projected/0a957598-1727-4fec-b5ac-29a407ec8c11-kube-api-access-9sqzk\") pod \"maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.729848 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.729681 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a957598-1727-4fec-b5ac-29a407ec8c11-openshift-service-ca\") pod \"maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.830740 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.830673 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a957598-1727-4fec-b5ac-29a407ec8c11-maas-api-tls\") pod \"maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.830946 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.830757 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqzk\" (UniqueName: \"kubernetes.io/projected/0a957598-1727-4fec-b5ac-29a407ec8c11-kube-api-access-9sqzk\") pod \"maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.830946 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.830790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a957598-1727-4fec-b5ac-29a407ec8c11-openshift-service-ca\") pod \"maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.831537 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.831456 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a957598-1727-4fec-b5ac-29a407ec8c11-openshift-service-ca\") pod \"maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.833348 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.833323 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a957598-1727-4fec-b5ac-29a407ec8c11-maas-api-tls\") pod \"maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.841787 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.841768 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqzk\" (UniqueName: \"kubernetes.io/projected/0a957598-1727-4fec-b5ac-29a407ec8c11-kube-api-access-9sqzk\") pod \"maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:40.932893 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:40.932795 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:41.087430 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:41.087401 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88"] Jul 24 16:13:41.088779 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:13:41.088751 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a957598_1727_4fec_b5ac_29a407ec8c11.slice/crio-fc40cf21b04d4ab2c3aeb85bf606928f7a31b205f24b0cc71c8e8e4b68a68888 WatchSource:0}: Error finding container fc40cf21b04d4ab2c3aeb85bf606928f7a31b205f24b0cc71c8e8e4b68a68888: Status 404 returned error can't find the container with id fc40cf21b04d4ab2c3aeb85bf606928f7a31b205f24b0cc71c8e8e4b68a68888 Jul 24 16:13:41.125599 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:41.125566 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" event={"ID":"0a957598-1727-4fec-b5ac-29a407ec8c11","Type":"ContainerStarted","Data":"fc40cf21b04d4ab2c3aeb85bf606928f7a31b205f24b0cc71c8e8e4b68a68888"} Jul 24 16:13:42.133229 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:42.133194 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" event={"ID":"0a957598-1727-4fec-b5ac-29a407ec8c11","Type":"ContainerStarted","Data":"0885588b97025967823251cbed0486821344cc08622c988d1d8c6f3f4bdbef01"} Jul 24 16:13:42.133649 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:42.133277 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:13:42.155387 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:42.155333 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" podStartSLOduration=1.7447481489999999 podStartE2EDuration="2.155312624s" totalImagesPullingTime="410.564475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:13:42.15434063 +0000 UTC m=+2263.387068320" watchObservedRunningTime="2026-07-24 16:13:42.155312624 +0000 UTC m=+2263.388040304" Jul 24 16:13:48.144004 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:13:48.143975 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:14:08.869163 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:08.869129 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc"] Jul 24 16:14:08.869686 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:08.869338 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" podUID="6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" containerName="maas-api" containerID="cri-o://9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7" gracePeriod=30 Jul 24 16:14:09.122187 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.122122 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:14:09.177337 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.177302 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-openshift-service-ca\") pod \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " Jul 24 16:14:09.177525 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.177368 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl7zs\" (UniqueName: \"kubernetes.io/projected/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-kube-api-access-fl7zs\") pod \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " Jul 24 16:14:09.177525 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.177430 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-maas-api-tls\") pod \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\" (UID: \"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3\") " Jul 24 16:14:09.177672 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.177650 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" (UID: "6d33cb02-cde0-42df-93ec-9a3b31eb8fa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:14:09.179727 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.179669 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" (UID: "6d33cb02-cde0-42df-93ec-9a3b31eb8fa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:14:09.179833 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.179723 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-kube-api-access-fl7zs" (OuterVolumeSpecName: "kube-api-access-fl7zs") pod "6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" (UID: "6d33cb02-cde0-42df-93ec-9a3b31eb8fa3"). InnerVolumeSpecName "kube-api-access-fl7zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:14:09.277866 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.277829 2571 generic.go:356] "Generic (PLEG): container finished" podID="6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" containerID="9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7" exitCode=0 Jul 24 16:14:09.278061 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.277878 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" event={"ID":"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3","Type":"ContainerDied","Data":"9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7"} Jul 24 16:14:09.278061 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.277944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" event={"ID":"6d33cb02-cde0-42df-93ec-9a3b31eb8fa3","Type":"ContainerDied","Data":"43432c202645d93f73c2f1da49b771865d92e41793973f6a4ede4e0183724bab"} Jul 24 16:14:09.278061 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.277971 2571 scope.go:117] "RemoveContainer" containerID="9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7" Jul 24 16:14:09.278061 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.277890 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" Jul 24 16:14:09.278291 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.278232 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl7zs\" (UniqueName: \"kubernetes.io/projected/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-kube-api-access-fl7zs\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:14:09.278291 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.278255 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:14:09.278291 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.278271 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:14:09.288151 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.288131 2571 scope.go:117] "RemoveContainer" containerID="9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7" Jul 24 16:14:09.288464 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:14:09.288427 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7\": container with ID starting with 9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7 not found: ID does not exist" containerID="9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7" Jul 24 16:14:09.288594 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.288470 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7"} err="failed to get container status \"9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7\": rpc error: code = NotFound desc = could not find container \"9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7\": container with ID starting with 9ae11feefdf3b5a3b579bf2dec9df05943f442e13951bd9eed6341ca0de41cd7 not found: ID does not exist" Jul 24 16:14:09.303113 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.303083 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc"] Jul 24 16:14:09.309075 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:09.309053 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc"] Jul 24 16:14:11.250769 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:11.250733 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" path="/var/lib/kubelet/pods/6d33cb02-cde0-42df-93ec-9a3b31eb8fa3/volumes" Jul 24 16:14:12.086352 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:12.086298 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-57ceec-764495b6d6-cxxzc" podUID="6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.59:8443/health\": context deadline exceeded" Jul 24 16:14:36.191239 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.191179 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88"] Jul 24 16:14:36.191847 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.191459 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" podUID="0a957598-1727-4fec-b5ac-29a407ec8c11" containerName="maas-api" containerID="cri-o://0885588b97025967823251cbed0486821344cc08622c988d1d8c6f3f4bdbef01" gracePeriod=30 Jul 24 16:14:36.434940 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.434909 2571 generic.go:356] "Generic (PLEG): container finished" podID="0a957598-1727-4fec-b5ac-29a407ec8c11" containerID="0885588b97025967823251cbed0486821344cc08622c988d1d8c6f3f4bdbef01" exitCode=0 Jul 24 16:14:36.435062 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.434980 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" event={"ID":"0a957598-1727-4fec-b5ac-29a407ec8c11","Type":"ContainerDied","Data":"0885588b97025967823251cbed0486821344cc08622c988d1d8c6f3f4bdbef01"} Jul 24 16:14:36.435062 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.435030 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" event={"ID":"0a957598-1727-4fec-b5ac-29a407ec8c11","Type":"ContainerDied","Data":"fc40cf21b04d4ab2c3aeb85bf606928f7a31b205f24b0cc71c8e8e4b68a68888"} Jul 24 16:14:36.435062 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.435046 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc40cf21b04d4ab2c3aeb85bf606928f7a31b205f24b0cc71c8e8e4b68a68888" Jul 24 16:14:36.444648 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.444596 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:14:36.529206 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.529166 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a957598-1727-4fec-b5ac-29a407ec8c11-openshift-service-ca\") pod \"0a957598-1727-4fec-b5ac-29a407ec8c11\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " Jul 24 16:14:36.529206 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.529213 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a957598-1727-4fec-b5ac-29a407ec8c11-maas-api-tls\") pod \"0a957598-1727-4fec-b5ac-29a407ec8c11\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " Jul 24 16:14:36.529433 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.529237 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sqzk\" (UniqueName: \"kubernetes.io/projected/0a957598-1727-4fec-b5ac-29a407ec8c11-kube-api-access-9sqzk\") pod \"0a957598-1727-4fec-b5ac-29a407ec8c11\" (UID: \"0a957598-1727-4fec-b5ac-29a407ec8c11\") " Jul 24 16:14:36.529553 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.529531 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a957598-1727-4fec-b5ac-29a407ec8c11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a957598-1727-4fec-b5ac-29a407ec8c11" (UID: "0a957598-1727-4fec-b5ac-29a407ec8c11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:14:36.531555 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.531533 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a957598-1727-4fec-b5ac-29a407ec8c11-kube-api-access-9sqzk" (OuterVolumeSpecName: "kube-api-access-9sqzk") pod "0a957598-1727-4fec-b5ac-29a407ec8c11" (UID: "0a957598-1727-4fec-b5ac-29a407ec8c11"). InnerVolumeSpecName "kube-api-access-9sqzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:14:36.531736 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.531685 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a957598-1727-4fec-b5ac-29a407ec8c11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a957598-1727-4fec-b5ac-29a407ec8c11" (UID: "0a957598-1727-4fec-b5ac-29a407ec8c11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:14:36.630508 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.630469 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a957598-1727-4fec-b5ac-29a407ec8c11-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:14:36.630508 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.630503 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a957598-1727-4fec-b5ac-29a407ec8c11-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:14:36.630508 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:36.630513 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sqzk\" (UniqueName: \"kubernetes.io/projected/0a957598-1727-4fec-b5ac-29a407ec8c11-kube-api-access-9sqzk\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:14:37.443993 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:37.443957 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88" Jul 24 16:14:37.462525 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:37.462489 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88"] Jul 24 16:14:37.467968 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:37.467941 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-afdeb1-7bfbb75cb4-2jj88"] Jul 24 16:14:39.251123 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:39.251084 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a957598-1727-4fec-b5ac-29a407ec8c11" path="/var/lib/kubelet/pods/0a957598-1727-4fec-b5ac-29a407ec8c11/volumes" Jul 24 16:14:51.852788 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.852681 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh"] Jul 24 16:14:51.853353 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.853320 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" containerName="maas-api" Jul 24 16:14:51.853353 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.853341 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" containerName="maas-api" Jul 24 16:14:51.853742 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.853364 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a957598-1727-4fec-b5ac-29a407ec8c11" containerName="maas-api" Jul 24 16:14:51.853742 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.853374 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a957598-1727-4fec-b5ac-29a407ec8c11" containerName="maas-api" Jul 24 16:14:51.853742 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.853451 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a957598-1727-4fec-b5ac-29a407ec8c11" containerName="maas-api" Jul 24 16:14:51.853742 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.853460 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d33cb02-cde0-42df-93ec-9a3b31eb8fa3" containerName="maas-api" Jul 24 16:14:51.857917 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.857896 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:51.860305 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.860284 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fafc28-serving-cert\"" Jul 24 16:14:51.867305 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.867282 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh"] Jul 24 16:14:51.958938 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.958897 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226d1a4e-025d-402b-a562-06308ef25162-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:51.959127 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.958951 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226d1a4e-025d-402b-a562-06308ef25162-maas-api-tls\") pod \"maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:51.959127 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:51.959027 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbd6\" (UniqueName: \"kubernetes.io/projected/226d1a4e-025d-402b-a562-06308ef25162-kube-api-access-fdbd6\") pod \"maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:52.059666 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:52.059630 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226d1a4e-025d-402b-a562-06308ef25162-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:52.059666 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:52.059671 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226d1a4e-025d-402b-a562-06308ef25162-maas-api-tls\") pod \"maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:52.059925 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:52.059717 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbd6\" (UniqueName: \"kubernetes.io/projected/226d1a4e-025d-402b-a562-06308ef25162-kube-api-access-fdbd6\") pod \"maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:52.060307 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:52.060288 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226d1a4e-025d-402b-a562-06308ef25162-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:52.062440 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:52.062417 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226d1a4e-025d-402b-a562-06308ef25162-maas-api-tls\") pod \"maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:52.073012 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:52.072987 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbd6\" (UniqueName: \"kubernetes.io/projected/226d1a4e-025d-402b-a562-06308ef25162-kube-api-access-fdbd6\") pod \"maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:52.172458 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:52.172379 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:52.314247 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:52.314068 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh"] Jul 24 16:14:52.316581 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:14:52.316552 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226d1a4e_025d_402b_a562_06308ef25162.slice/crio-a32f79f19fcf848f433bf77f8f1b85e783cd5e3bdd90d336ff82ced6cee1510b WatchSource:0}: Error finding container a32f79f19fcf848f433bf77f8f1b85e783cd5e3bdd90d336ff82ced6cee1510b: Status 404 returned error can't find the container with id a32f79f19fcf848f433bf77f8f1b85e783cd5e3bdd90d336ff82ced6cee1510b Jul 24 16:14:52.524668 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:52.524636 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" event={"ID":"226d1a4e-025d-402b-a562-06308ef25162","Type":"ContainerStarted","Data":"a32f79f19fcf848f433bf77f8f1b85e783cd5e3bdd90d336ff82ced6cee1510b"} Jul 24 16:14:53.531416 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:53.531378 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" event={"ID":"226d1a4e-025d-402b-a562-06308ef25162","Type":"ContainerStarted","Data":"ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da"} Jul 24 16:14:53.531875 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:53.531482 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:14:53.551168 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:53.551100 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" podStartSLOduration=2.099480544 podStartE2EDuration="2.551082861s" totalImagesPullingTime="451.602317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:14:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:14:53.549490248 +0000 UTC m=+2334.782217939" watchObservedRunningTime="2026-07-24 16:14:53.551082861 +0000 UTC m=+2334.783810541" Jul 24 16:14:59.544882 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:14:59.544853 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:15:00.318753 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.318716 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g"] Jul 24 16:15:00.326101 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.326074 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.329421 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.329394 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c6c91e-serving-cert\"" Jul 24 16:15:00.334176 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.334078 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g"] Jul 24 16:15:00.431709 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.431659 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.431893 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.431746 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.431893 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.431803 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6fzx\" (UniqueName: \"kubernetes.io/projected/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-kube-api-access-r6fzx\") pod \"maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.532313 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.532276 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.532495 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.532342 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6fzx\" (UniqueName: \"kubernetes.io/projected/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-kube-api-access-r6fzx\") pod \"maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.532495 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.532419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.533104 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.533087 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.534889 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.534865 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-maas-api-tls\") pod \"maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.542927 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.542899 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6fzx\" (UniqueName: \"kubernetes.io/projected/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-kube-api-access-r6fzx\") pod \"maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.639511 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.639421 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:00.778756 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:00.778724 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g"] Jul 24 16:15:00.780156 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:15:00.780134 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdbe9e1_63e6_430e_bcdb_fda3502aaf55.slice/crio-eaefce2375701c2c4b5e645ce7f080a038830c83ea522fe0b33fa848a906456c WatchSource:0}: Error finding container eaefce2375701c2c4b5e645ce7f080a038830c83ea522fe0b33fa848a906456c: Status 404 returned error can't find the container with id eaefce2375701c2c4b5e645ce7f080a038830c83ea522fe0b33fa848a906456c Jul 24 16:15:01.576453 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:01.576422 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" event={"ID":"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55","Type":"ContainerStarted","Data":"eaefce2375701c2c4b5e645ce7f080a038830c83ea522fe0b33fa848a906456c"} Jul 24 16:15:02.583676 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:02.583641 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" event={"ID":"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55","Type":"ContainerStarted","Data":"f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564"} Jul 24 16:15:02.584118 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:02.583722 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:02.607405 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:02.607356 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" podStartSLOduration=1.8548208050000001 podStartE2EDuration="2.607341588s" totalImagesPullingTime="752.520783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:15:02.606934298 +0000 UTC m=+2343.839661976" watchObservedRunningTime="2026-07-24 16:15:02.607341588 +0000 UTC m=+2343.840069267" Jul 24 16:15:08.594154 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:08.594125 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:15:20.009056 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.009025 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g"] Jul 24 16:15:20.012549 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.012520 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.015434 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.015411 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fafc28\"/\"default-dockercfg-pzqn4\"" Jul 24 16:15:20.016510 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.016482 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fafc28\"/\"auth-test-model-fafc28-kserve-self-signed-certs\"" Jul 24 16:15:20.016636 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.016553 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fafc28\"/\"kube-root-ca.crt\"" Jul 24 16:15:20.016636 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.016581 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fafc28\"/\"openshift-service-ca.crt\"" Jul 24 16:15:20.025897 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.025875 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g"] Jul 24 16:15:20.109356 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.109315 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-model-cache\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.109566 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.109373 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cb4694-6055-43cd-89bf-58c97518557c-tls-certs\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.109566 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.109407 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmsh\" (UniqueName: \"kubernetes.io/projected/66cb4694-6055-43cd-89bf-58c97518557c-kube-api-access-9pmsh\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.109566 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.109493 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-dshm\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.109566 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.109521 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-tmp-dir\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.109566 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.109551 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-home\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.209988 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.209954 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-model-cache\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.210216 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.210012 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cb4694-6055-43cd-89bf-58c97518557c-tls-certs\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.210216 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.210042 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmsh\" (UniqueName: \"kubernetes.io/projected/66cb4694-6055-43cd-89bf-58c97518557c-kube-api-access-9pmsh\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.210216 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.210077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-dshm\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.210216 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.210101 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-tmp-dir\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.210216 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.210136 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-home\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.210506 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.210471 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-model-cache\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.210574 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.210513 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-home\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.210832 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.210802 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-tmp-dir\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.213262 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.213224 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cb4694-6055-43cd-89bf-58c97518557c-tls-certs\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.213379 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.213231 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-dshm\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.219072 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.219050 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmsh\" (UniqueName: \"kubernetes.io/projected/66cb4694-6055-43cd-89bf-58c97518557c-kube-api-access-9pmsh\") pod \"auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.322989 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.322893 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:20.465571 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.465518 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g"] Jul 24 16:15:20.467873 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:15:20.467844 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66cb4694_6055_43cd_89bf_58c97518557c.slice/crio-fc0439cd22c63acb2f797318c1714ddda78c937f7c3eb37f2c980a26c3e87e36 WatchSource:0}: Error finding container fc0439cd22c63acb2f797318c1714ddda78c937f7c3eb37f2c980a26c3e87e36: Status 404 returned error can't find the container with id fc0439cd22c63acb2f797318c1714ddda78c937f7c3eb37f2c980a26c3e87e36 Jul 24 16:15:20.691760 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.691650 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" event={"ID":"66cb4694-6055-43cd-89bf-58c97518557c","Type":"ContainerStarted","Data":"fffb39c44acadfde2b0e013e5bbf0b77686cb4d4e5ad89133e2da338ee9dbec5"} Jul 24 16:15:20.691760 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.691713 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" event={"ID":"66cb4694-6055-43cd-89bf-58c97518557c","Type":"ContainerStarted","Data":"fc0439cd22c63acb2f797318c1714ddda78c937f7c3eb37f2c980a26c3e87e36"} Jul 24 16:15:20.715468 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:20.715406 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" podStartSLOduration=1.7153835160000002 podStartE2EDuration="1.715383516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:15:20.714998044 +0000 UTC m=+2361.947725723" watchObservedRunningTime="2026-07-24 16:15:20.715383516 +0000 UTC m=+2361.948111195" Jul 24 16:15:30.323384 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:30.323343 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:30.323851 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:30.323629 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:30.336340 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:30.336314 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:30.756159 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:30.756130 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:15:59.388452 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:59.388422 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:15:59.393137 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:59.393116 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:15:59.394222 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:59.394204 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:15:59.398684 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:15:59.398666 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:18:04.483183 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:04.483088 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g"] Jul 24 16:18:19.505120 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.505074 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" podUID="66cb4694-6055-43cd-89bf-58c97518557c" containerName="main" containerID="cri-o://fffb39c44acadfde2b0e013e5bbf0b77686cb4d4e5ad89133e2da338ee9dbec5" gracePeriod=45 Jul 24 16:18:19.696505 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.696466 2571 generic.go:356] "Generic (PLEG): container finished" podID="66cb4694-6055-43cd-89bf-58c97518557c" containerID="fffb39c44acadfde2b0e013e5bbf0b77686cb4d4e5ad89133e2da338ee9dbec5" exitCode=0 Jul 24 16:18:19.696670 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.696560 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" event={"ID":"66cb4694-6055-43cd-89bf-58c97518557c","Type":"ContainerDied","Data":"fffb39c44acadfde2b0e013e5bbf0b77686cb4d4e5ad89133e2da338ee9dbec5"} Jul 24 16:18:19.772363 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.772340 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:18:19.870970 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.870937 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-tmp-dir\") pod \"66cb4694-6055-43cd-89bf-58c97518557c\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " Jul 24 16:18:19.871153 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871005 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-home\") pod \"66cb4694-6055-43cd-89bf-58c97518557c\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " Jul 24 16:18:19.871153 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871030 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-model-cache\") pod \"66cb4694-6055-43cd-89bf-58c97518557c\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " Jul 24 16:18:19.871153 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871051 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-dshm\") pod \"66cb4694-6055-43cd-89bf-58c97518557c\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " Jul 24 16:18:19.871153 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871105 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cb4694-6055-43cd-89bf-58c97518557c-tls-certs\") pod \"66cb4694-6055-43cd-89bf-58c97518557c\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " Jul 24 16:18:19.871153 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871141 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pmsh\" (UniqueName: \"kubernetes.io/projected/66cb4694-6055-43cd-89bf-58c97518557c-kube-api-access-9pmsh\") pod \"66cb4694-6055-43cd-89bf-58c97518557c\" (UID: \"66cb4694-6055-43cd-89bf-58c97518557c\") " Jul 24 16:18:19.871432 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871197 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66cb4694-6055-43cd-89bf-58c97518557c" (UID: "66cb4694-6055-43cd-89bf-58c97518557c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:18:19.871432 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871290 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-home" (OuterVolumeSpecName: "home") pod "66cb4694-6055-43cd-89bf-58c97518557c" (UID: "66cb4694-6055-43cd-89bf-58c97518557c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:18:19.871432 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871304 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-model-cache" (OuterVolumeSpecName: "model-cache") pod "66cb4694-6055-43cd-89bf-58c97518557c" (UID: "66cb4694-6055-43cd-89bf-58c97518557c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:18:19.871554 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871454 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-home\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:18:19.871554 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871473 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-model-cache\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:18:19.871554 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.871507 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-tmp-dir\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:18:19.873574 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.873545 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-dshm" (OuterVolumeSpecName: "dshm") pod "66cb4694-6055-43cd-89bf-58c97518557c" (UID: "66cb4694-6055-43cd-89bf-58c97518557c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:18:19.873736 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.873591 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cb4694-6055-43cd-89bf-58c97518557c-kube-api-access-9pmsh" (OuterVolumeSpecName: "kube-api-access-9pmsh") pod "66cb4694-6055-43cd-89bf-58c97518557c" (UID: "66cb4694-6055-43cd-89bf-58c97518557c"). InnerVolumeSpecName "kube-api-access-9pmsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:18:19.873736 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.873588 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66cb4694-6055-43cd-89bf-58c97518557c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66cb4694-6055-43cd-89bf-58c97518557c" (UID: "66cb4694-6055-43cd-89bf-58c97518557c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:18:19.972333 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.972300 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66cb4694-6055-43cd-89bf-58c97518557c-dshm\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:18:19.972333 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.972329 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cb4694-6055-43cd-89bf-58c97518557c-tls-certs\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:18:19.972333 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:19.972340 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmsh\" (UniqueName: \"kubernetes.io/projected/66cb4694-6055-43cd-89bf-58c97518557c-kube-api-access-9pmsh\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:18:20.703096 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:20.703071 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" Jul 24 16:18:20.703096 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:20.703078 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g" event={"ID":"66cb4694-6055-43cd-89bf-58c97518557c","Type":"ContainerDied","Data":"fc0439cd22c63acb2f797318c1714ddda78c937f7c3eb37f2c980a26c3e87e36"} Jul 24 16:18:20.703628 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:20.703123 2571 scope.go:117] "RemoveContainer" containerID="fffb39c44acadfde2b0e013e5bbf0b77686cb4d4e5ad89133e2da338ee9dbec5" Jul 24 16:18:20.741005 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:20.740970 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g"] Jul 24 16:18:20.744972 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:20.744945 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/auth-test-model-fafc28-kserve-dfb7dcf9f-bx62g"] Jul 24 16:18:21.251504 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:18:21.251473 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66cb4694-6055-43cd-89bf-58c97518557c" path="/var/lib/kubelet/pods/66cb4694-6055-43cd-89bf-58c97518557c/volumes" Jul 24 16:19:43.775103 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.775018 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4"] Jul 24 16:19:43.775602 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.775387 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66cb4694-6055-43cd-89bf-58c97518557c" containerName="main" Jul 24 16:19:43.775602 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.775399 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="66cb4694-6055-43cd-89bf-58c97518557c" containerName="main" Jul 24 16:19:43.775602 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.775475 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="66cb4694-6055-43cd-89bf-58c97518557c" containerName="main" Jul 24 16:19:43.778447 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.778429 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:43.781926 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.781897 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fafc28\"/\"openshift-service-ca.crt\"" Jul 24 16:19:43.782322 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.782121 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fafc28\"/\"kube-root-ca.crt\"" Jul 24 16:19:43.783344 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.783324 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fafc28\"/\"rate-test-model-fafc28-kserve-self-signed-certs\"" Jul 24 16:19:43.783452 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.783348 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fafc28\"/\"default-dockercfg-pzqn4\"" Jul 24 16:19:43.795172 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.795141 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4"] Jul 24 16:19:43.937114 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.937075 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-dshm\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:43.937114 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.937113 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5nrr\" (UniqueName: \"kubernetes.io/projected/81e28766-073a-4984-9c87-ebe8a8ac196a-kube-api-access-c5nrr\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:43.937343 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.937136 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-home\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:43.937343 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.937223 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-tmp-dir\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:43.937343 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.937255 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e28766-073a-4984-9c87-ebe8a8ac196a-tls-certs\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:43.937343 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:43.937283 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-model-cache\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.037893 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.037803 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-home\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.037893 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.037874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-tmp-dir\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.038112 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.037914 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e28766-073a-4984-9c87-ebe8a8ac196a-tls-certs\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.038112 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.037944 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-model-cache\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.038112 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.037982 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-dshm\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.038112 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.038009 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5nrr\" (UniqueName: \"kubernetes.io/projected/81e28766-073a-4984-9c87-ebe8a8ac196a-kube-api-access-c5nrr\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.038274 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.038257 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-home\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.038323 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.038305 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-tmp-dir\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.038382 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.038321 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-model-cache\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.040453 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.040423 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-dshm\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.040650 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.040631 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e28766-073a-4984-9c87-ebe8a8ac196a-tls-certs\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.050179 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.050156 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5nrr\" (UniqueName: \"kubernetes.io/projected/81e28766-073a-4984-9c87-ebe8a8ac196a-kube-api-access-c5nrr\") pod \"rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.089942 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.089903 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:44.236281 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:44.236255 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4"] Jul 24 16:19:45.172842 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:45.172801 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" event={"ID":"81e28766-073a-4984-9c87-ebe8a8ac196a","Type":"ContainerStarted","Data":"6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426"} Jul 24 16:19:45.173272 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:45.172855 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" event={"ID":"81e28766-073a-4984-9c87-ebe8a8ac196a","Type":"ContainerStarted","Data":"b1e1dc362fde9df45e6843c54c11b4d58d5feb4757d4f793e816fac08645677a"} Jul 24 16:19:45.195881 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:45.195833 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" podStartSLOduration=2.195816192 podStartE2EDuration="2.195816192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:19:45.1950532 +0000 UTC m=+2626.427780879" watchObservedRunningTime="2026-07-24 16:19:45.195816192 +0000 UTC m=+2626.428543870" Jul 24 16:19:54.090380 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:54.090338 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:54.090985 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:54.090404 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:54.103236 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:54.103204 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:19:54.244632 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:19:54.244599 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:20:00.007400 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:20:00.007369 2571 scope.go:117] "RemoveContainer" containerID="0885588b97025967823251cbed0486821344cc08622c988d1d8c6f3f4bdbef01" Jul 24 16:20:52.704769 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:20:52.704663 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4"] Jul 24 16:20:59.432097 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:20:59.432058 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:20:59.437310 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:20:59.437284 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:20:59.437495 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:20:59.437481 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:20:59.442657 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:20:59.442632 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:21:07.724423 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:07.724375 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" podUID="81e28766-073a-4984-9c87-ebe8a8ac196a" containerName="main" containerID="cri-o://6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426" gracePeriod=45 Jul 24 16:21:07.983271 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:07.983205 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:21:08.089219 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089185 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-dshm\") pod \"81e28766-073a-4984-9c87-ebe8a8ac196a\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " Jul 24 16:21:08.089437 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089254 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-model-cache\") pod \"81e28766-073a-4984-9c87-ebe8a8ac196a\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " Jul 24 16:21:08.089437 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089276 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-tmp-dir\") pod \"81e28766-073a-4984-9c87-ebe8a8ac196a\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " Jul 24 16:21:08.089437 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089301 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e28766-073a-4984-9c87-ebe8a8ac196a-tls-certs\") pod \"81e28766-073a-4984-9c87-ebe8a8ac196a\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " Jul 24 16:21:08.089437 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089352 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5nrr\" (UniqueName: \"kubernetes.io/projected/81e28766-073a-4984-9c87-ebe8a8ac196a-kube-api-access-c5nrr\") pod \"81e28766-073a-4984-9c87-ebe8a8ac196a\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " Jul 24 16:21:08.089437 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089412 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-home\") pod \"81e28766-073a-4984-9c87-ebe8a8ac196a\" (UID: \"81e28766-073a-4984-9c87-ebe8a8ac196a\") " Jul 24 16:21:08.089742 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089593 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81e28766-073a-4984-9c87-ebe8a8ac196a" (UID: "81e28766-073a-4984-9c87-ebe8a8ac196a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:21:08.089742 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089605 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-model-cache" (OuterVolumeSpecName: "model-cache") pod "81e28766-073a-4984-9c87-ebe8a8ac196a" (UID: "81e28766-073a-4984-9c87-ebe8a8ac196a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:21:08.089829 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089760 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-model-cache\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:21:08.089829 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089782 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-tmp-dir\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:21:08.089923 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.089894 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-home" (OuterVolumeSpecName: "home") pod "81e28766-073a-4984-9c87-ebe8a8ac196a" (UID: "81e28766-073a-4984-9c87-ebe8a8ac196a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:21:08.091599 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.091580 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-dshm" (OuterVolumeSpecName: "dshm") pod "81e28766-073a-4984-9c87-ebe8a8ac196a" (UID: "81e28766-073a-4984-9c87-ebe8a8ac196a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:21:08.091728 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.091686 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e28766-073a-4984-9c87-ebe8a8ac196a-kube-api-access-c5nrr" (OuterVolumeSpecName: "kube-api-access-c5nrr") pod "81e28766-073a-4984-9c87-ebe8a8ac196a" (UID: "81e28766-073a-4984-9c87-ebe8a8ac196a"). InnerVolumeSpecName "kube-api-access-c5nrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:21:08.091728 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.091712 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81e28766-073a-4984-9c87-ebe8a8ac196a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81e28766-073a-4984-9c87-ebe8a8ac196a" (UID: "81e28766-073a-4984-9c87-ebe8a8ac196a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:21:08.191301 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.191262 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5nrr\" (UniqueName: \"kubernetes.io/projected/81e28766-073a-4984-9c87-ebe8a8ac196a-kube-api-access-c5nrr\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:21:08.191301 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.191297 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-home\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:21:08.191301 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.191312 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81e28766-073a-4984-9c87-ebe8a8ac196a-dshm\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:21:08.191544 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.191324 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81e28766-073a-4984-9c87-ebe8a8ac196a-tls-certs\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:21:08.643737 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.643676 2571 generic.go:356] "Generic (PLEG): container finished" podID="81e28766-073a-4984-9c87-ebe8a8ac196a" containerID="6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426" exitCode=0 Jul 24 16:21:08.643737 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.643731 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" Jul 24 16:21:08.643950 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.643762 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" event={"ID":"81e28766-073a-4984-9c87-ebe8a8ac196a","Type":"ContainerDied","Data":"6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426"} Jul 24 16:21:08.643950 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.643811 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4" event={"ID":"81e28766-073a-4984-9c87-ebe8a8ac196a","Type":"ContainerDied","Data":"b1e1dc362fde9df45e6843c54c11b4d58d5feb4757d4f793e816fac08645677a"} Jul 24 16:21:08.643950 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.643830 2571 scope.go:117] "RemoveContainer" containerID="6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426" Jul 24 16:21:08.654119 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.654094 2571 scope.go:117] "RemoveContainer" containerID="6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426" Jul 24 16:21:08.654390 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:21:08.654373 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426\": container with ID starting with 6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426 not found: ID does not exist" containerID="6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426" Jul 24 16:21:08.654441 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.654398 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426"} err="failed to get container status \"6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426\": rpc error: code = NotFound desc = could not find container \"6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426\": container with ID starting with 6b157023604f120ae7d64c654716ab8204843fdbb1b6172ea6b274c0ac7e7426 not found: ID does not exist" Jul 24 16:21:08.675173 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.675128 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4"] Jul 24 16:21:08.680878 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:08.680851 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fafc28/rate-test-model-fafc28-kserve-c8fdc7b9f-qn2c4"] Jul 24 16:21:09.251156 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:09.251117 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81e28766-073a-4984-9c87-ebe8a8ac196a" path="/var/lib/kubelet/pods/81e28766-073a-4984-9c87-ebe8a8ac196a/volumes" Jul 24 16:21:10.280355 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.280323 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7484-140031903270352"] Jul 24 16:21:10.280787 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.280739 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81e28766-073a-4984-9c87-ebe8a8ac196a" containerName="main" Jul 24 16:21:10.280787 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.280751 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="81e28766-073a-4984-9c87-ebe8a8ac196a" containerName="main" Jul 24 16:21:10.280864 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.280815 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="81e28766-073a-4984-9c87-ebe8a8ac196a" containerName="main" Jul 24 16:21:10.285233 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.285214 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-140031903270352" Jul 24 16:21:10.288051 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.288018 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xzxkn\"" Jul 24 16:21:10.290270 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.290241 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-140031903270352"] Jul 24 16:21:10.412888 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.412853 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffcst\" (UniqueName: \"kubernetes.io/projected/b62140e0-fed4-4fec-b757-4762d3b9623f-kube-api-access-ffcst\") pod \"test-curl-7484-140031903270352\" (UID: \"b62140e0-fed4-4fec-b757-4762d3b9623f\") " pod="opendatahub/test-curl-7484-140031903270352" Jul 24 16:21:10.514393 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.514357 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffcst\" (UniqueName: \"kubernetes.io/projected/b62140e0-fed4-4fec-b757-4762d3b9623f-kube-api-access-ffcst\") pod \"test-curl-7484-140031903270352\" (UID: \"b62140e0-fed4-4fec-b757-4762d3b9623f\") " pod="opendatahub/test-curl-7484-140031903270352" Jul 24 16:21:10.523409 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.523385 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffcst\" (UniqueName: \"kubernetes.io/projected/b62140e0-fed4-4fec-b757-4762d3b9623f-kube-api-access-ffcst\") pod \"test-curl-7484-140031903270352\" (UID: \"b62140e0-fed4-4fec-b757-4762d3b9623f\") " pod="opendatahub/test-curl-7484-140031903270352" Jul 24 16:21:10.596282 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.596202 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-140031903270352" Jul 24 16:21:10.743237 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.743081 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7484-140031903270352"] Jul 24 16:21:10.746120 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:21:10.746089 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb62140e0_fed4_4fec_b757_4762d3b9623f.slice/crio-3650f471ab8524df457ec9244340e523ecb59e5dabcab2c8568f895b475d2bd8 WatchSource:0}: Error finding container 3650f471ab8524df457ec9244340e523ecb59e5dabcab2c8568f895b475d2bd8: Status 404 returned error can't find the container with id 3650f471ab8524df457ec9244340e523ecb59e5dabcab2c8568f895b475d2bd8 Jul 24 16:21:10.747464 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:10.747449 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:21:11.666651 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:11.666619 2571 generic.go:356] "Generic (PLEG): container finished" podID="b62140e0-fed4-4fec-b757-4762d3b9623f" containerID="d8cf5235d4abaf87188d4f5e5400f2c7168c08a3d0b338210345225f94c93720" exitCode=0 Jul 24 16:21:11.667060 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:11.666744 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-140031903270352" event={"ID":"b62140e0-fed4-4fec-b757-4762d3b9623f","Type":"ContainerDied","Data":"d8cf5235d4abaf87188d4f5e5400f2c7168c08a3d0b338210345225f94c93720"} Jul 24 16:21:11.667060 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:11.666788 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7484-140031903270352" event={"ID":"b62140e0-fed4-4fec-b757-4762d3b9623f","Type":"ContainerStarted","Data":"3650f471ab8524df457ec9244340e523ecb59e5dabcab2c8568f895b475d2bd8"} Jul 24 16:21:12.807915 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:12.807882 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-140031903270352" Jul 24 16:21:12.855315 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:12.855288 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7484-140031903270352_b62140e0-fed4-4fec-b757-4762d3b9623f/test-curl-7484-140031903270352/0.log" Jul 24 16:21:12.887237 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:12.887206 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7484-140031903270352"] Jul 24 16:21:12.897893 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:12.897873 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7484-140031903270352"] Jul 24 16:21:12.937855 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:12.937796 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ffcst\" (UniqueName: \"kubernetes.io/projected/b62140e0-fed4-4fec-b757-4762d3b9623f-kube-api-access-ffcst\") pod \"b62140e0-fed4-4fec-b757-4762d3b9623f\" (UID: \"b62140e0-fed4-4fec-b757-4762d3b9623f\") " Jul 24 16:21:12.939997 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:12.939966 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62140e0-fed4-4fec-b757-4762d3b9623f-kube-api-access-ffcst" (OuterVolumeSpecName: "kube-api-access-ffcst") pod "b62140e0-fed4-4fec-b757-4762d3b9623f" (UID: "b62140e0-fed4-4fec-b757-4762d3b9623f"). InnerVolumeSpecName "kube-api-access-ffcst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:21:13.038599 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:13.038564 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffcst\" (UniqueName: \"kubernetes.io/projected/b62140e0-fed4-4fec-b757-4762d3b9623f-kube-api-access-ffcst\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:21:13.251020 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:13.250978 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b62140e0-fed4-4fec-b757-4762d3b9623f" path="/var/lib/kubelet/pods/b62140e0-fed4-4fec-b757-4762d3b9623f/volumes" Jul 24 16:21:13.678538 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:13.678454 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7484-140031903270352" Jul 24 16:21:13.678538 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:13.678514 2571 scope.go:117] "RemoveContainer" containerID="d8cf5235d4abaf87188d4f5e5400f2c7168c08a3d0b338210345225f94c93720" Jul 24 16:21:17.993410 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:17.993366 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf"] Jul 24 16:21:17.993964 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:17.993944 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b62140e0-fed4-4fec-b757-4762d3b9623f" containerName="test-curl-7484-140031903270352" Jul 24 16:21:17.994047 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:17.993967 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b62140e0-fed4-4fec-b757-4762d3b9623f" containerName="test-curl-7484-140031903270352" Jul 24 16:21:17.994101 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:17.994063 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b62140e0-fed4-4fec-b757-4762d3b9623f" containerName="test-curl-7484-140031903270352" Jul 24 16:21:17.998592 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:17.998570 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.001104 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.001086 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-5d0582-serving-cert\"" Jul 24 16:21:18.009039 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.009008 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf"] Jul 24 16:21:18.089750 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.089716 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.089972 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.089846 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9sr\" (UniqueName: \"kubernetes.io/projected/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-kube-api-access-kf9sr\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.089972 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.089890 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.191298 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.191263 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.191510 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.191339 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9sr\" (UniqueName: \"kubernetes.io/projected/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-kube-api-access-kf9sr\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.191510 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.191373 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.191510 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:21:18.191460 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-serving-cert: secret "maas-api-e2e-ipp-a-5d0582-serving-cert" not found Jul 24 16:21:18.191726 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:21:18.191537 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-maas-api-tls podName:7b690939-7fc4-4e9e-bfb2-9a45633d2a5a nodeName:}" failed. No retries permitted until 2026-07-24 16:21:18.69151624 +0000 UTC m=+2719.924243909 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-maas-api-tls") pod "maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" (UID: "7b690939-7fc4-4e9e-bfb2-9a45633d2a5a") : secret "maas-api-e2e-ipp-a-5d0582-serving-cert" not found Jul 24 16:21:18.192084 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.192067 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.200558 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.200521 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9sr\" (UniqueName: \"kubernetes.io/projected/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-kube-api-access-kf9sr\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.697406 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.697366 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.700221 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.700198 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:18.910497 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:18.910447 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:19.048286 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:19.048260 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf"] Jul 24 16:21:19.049950 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:21:19.049919 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b690939_7fc4_4e9e_bfb2_9a45633d2a5a.slice/crio-45ed39d234995c857453ebafcb4bbbe897d14b2565aedf308fa244b7d44e398c WatchSource:0}: Error finding container 45ed39d234995c857453ebafcb4bbbe897d14b2565aedf308fa244b7d44e398c: Status 404 returned error can't find the container with id 45ed39d234995c857453ebafcb4bbbe897d14b2565aedf308fa244b7d44e398c Jul 24 16:21:19.721133 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:19.721032 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" event={"ID":"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a","Type":"ContainerStarted","Data":"1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c"} Jul 24 16:21:19.721133 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:19.721136 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:19.721411 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:19.721158 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" event={"ID":"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a","Type":"ContainerStarted","Data":"45ed39d234995c857453ebafcb4bbbe897d14b2565aedf308fa244b7d44e398c"} Jul 24 16:21:19.741911 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:19.741861 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" podStartSLOduration=2.2704427369999998 podStartE2EDuration="2.741844537s" totalImagesPullingTime="471.4018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:21:19.73997314 +0000 UTC m=+2720.972700818" watchObservedRunningTime="2026-07-24 16:21:19.741844537 +0000 UTC m=+2720.974572214" Jul 24 16:21:25.731939 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:25.731910 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:21:37.750435 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.750397 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv"] Jul 24 16:21:37.753934 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.753916 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:37.756544 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.756524 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-07e89c-serving-cert\"" Jul 24 16:21:37.768884 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.768861 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv"] Jul 24 16:21:37.770907 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.770876 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff39d7-0128-4186-aeb7-57754dcadde7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:37.771036 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.770964 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:37.771036 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.770993 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2vd9\" (UniqueName: \"kubernetes.io/projected/dfff39d7-0128-4186-aeb7-57754dcadde7-kube-api-access-h2vd9\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:37.871799 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.871764 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff39d7-0128-4186-aeb7-57754dcadde7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:37.872063 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.871821 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:37.872063 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.871842 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2vd9\" (UniqueName: \"kubernetes.io/projected/dfff39d7-0128-4186-aeb7-57754dcadde7-kube-api-access-h2vd9\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:37.872063 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:21:37.871943 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-serving-cert: secret "maas-api-e2e-ipp-b-07e89c-serving-cert" not found Jul 24 16:21:37.872063 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:21:37.872021 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls podName:dfff39d7-0128-4186-aeb7-57754dcadde7 nodeName:}" failed. No retries permitted until 2026-07-24 16:21:38.372000988 +0000 UTC m=+2739.604728645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls") pod "maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" (UID: "dfff39d7-0128-4186-aeb7-57754dcadde7") : secret "maas-api-e2e-ipp-b-07e89c-serving-cert" not found Jul 24 16:21:37.872405 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.872387 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff39d7-0128-4186-aeb7-57754dcadde7-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:37.883453 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:37.883420 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2vd9\" (UniqueName: \"kubernetes.io/projected/dfff39d7-0128-4186-aeb7-57754dcadde7-kube-api-access-h2vd9\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:38.376336 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:38.376303 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:38.376500 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:21:38.376458 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-serving-cert: secret "maas-api-e2e-ipp-b-07e89c-serving-cert" not found Jul 24 16:21:38.376575 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:21:38.376548 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls podName:dfff39d7-0128-4186-aeb7-57754dcadde7 nodeName:}" failed. No retries permitted until 2026-07-24 16:21:39.376529435 +0000 UTC m=+2740.609257093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls") pod "maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" (UID: "dfff39d7-0128-4186-aeb7-57754dcadde7") : secret "maas-api-e2e-ipp-b-07e89c-serving-cert" not found Jul 24 16:21:39.385042 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:39.385005 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:39.387709 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:39.387671 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls\") pod \"maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:39.564642 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:39.564609 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:39.715518 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:39.715493 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv"] Jul 24 16:21:39.716330 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:21:39.716304 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfff39d7_0128_4186_aeb7_57754dcadde7.slice/crio-9de4e45d41839b340868f4a99dd941e4475cbea40ff0efbeaac117849b2ca59b WatchSource:0}: Error finding container 9de4e45d41839b340868f4a99dd941e4475cbea40ff0efbeaac117849b2ca59b: Status 404 returned error can't find the container with id 9de4e45d41839b340868f4a99dd941e4475cbea40ff0efbeaac117849b2ca59b Jul 24 16:21:39.835244 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:39.835213 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" event={"ID":"dfff39d7-0128-4186-aeb7-57754dcadde7","Type":"ContainerStarted","Data":"9de4e45d41839b340868f4a99dd941e4475cbea40ff0efbeaac117849b2ca59b"} Jul 24 16:21:40.842966 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:40.842931 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" event={"ID":"dfff39d7-0128-4186-aeb7-57754dcadde7","Type":"ContainerStarted","Data":"fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8"} Jul 24 16:21:40.843376 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:40.843018 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:21:40.862840 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:40.862792 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" podStartSLOduration=3.49387792 podStartE2EDuration="3.862778817s" totalImagesPullingTime="368.900897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:21:40.860007534 +0000 UTC m=+2742.092735216" watchObservedRunningTime="2026-07-24 16:21:40.862778817 +0000 UTC m=+2742.095506495" Jul 24 16:21:46.854619 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:21:46.854582 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:22:46.216723 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.216617 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g"] Jul 24 16:22:46.220327 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.220310 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.222878 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.222857 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-ebf027-serving-cert\"" Jul 24 16:22:46.229830 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.229806 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g"] Jul 24 16:22:46.315785 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.315743 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf66c\" (UniqueName: \"kubernetes.io/projected/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-kube-api-access-wf66c\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.315993 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.315796 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.315993 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.315873 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.416959 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.416925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.417143 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.417027 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf66c\" (UniqueName: \"kubernetes.io/projected/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-kube-api-access-wf66c\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.417143 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.417054 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.417230 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:22:46.417149 2571 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-serving-cert: secret "maas-api-e2e-ipp-cleanup-ebf027-serving-cert" not found Jul 24 16:22:46.417230 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:22:46.417212 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-maas-api-tls podName:cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0 nodeName:}" failed. No retries permitted until 2026-07-24 16:22:46.917197033 +0000 UTC m=+2808.149924689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" (UID: "cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0") : secret "maas-api-e2e-ipp-cleanup-ebf027-serving-cert" not found Jul 24 16:22:46.417539 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.417523 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.425859 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.425838 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf66c\" (UniqueName: \"kubernetes.io/projected/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-kube-api-access-wf66c\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.923544 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.923503 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:46.926269 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:46.926237 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:47.132673 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:47.132627 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:47.288976 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:47.288942 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g"] Jul 24 16:22:47.292940 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:22:47.292913 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf17559c_c56e_4fb7_b1ec_1f1b3d8b67f0.slice/crio-dba99d81ec64b90f06ce232491cb1e016fed690d003870b6319a2c77981c9eff WatchSource:0}: Error finding container dba99d81ec64b90f06ce232491cb1e016fed690d003870b6319a2c77981c9eff: Status 404 returned error can't find the container with id dba99d81ec64b90f06ce232491cb1e016fed690d003870b6319a2c77981c9eff Jul 24 16:22:48.214981 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:48.214944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" event={"ID":"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0","Type":"ContainerStarted","Data":"958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de"} Jul 24 16:22:48.215152 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:48.215044 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" event={"ID":"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0","Type":"ContainerStarted","Data":"dba99d81ec64b90f06ce232491cb1e016fed690d003870b6319a2c77981c9eff"} Jul 24 16:22:48.215152 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:48.215072 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:22:48.232973 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:48.232926 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" podStartSLOduration=1.879032782 podStartE2EDuration="2.232911197s" totalImagesPullingTime="353.878415ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:22:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:22:48.232152998 +0000 UTC m=+2809.464880702" watchObservedRunningTime="2026-07-24 16:22:48.232911197 +0000 UTC m=+2809.465638874" Jul 24 16:22:54.226565 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:22:54.226533 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:23:26.888331 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:26.888270 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g"] Jul 24 16:23:26.889986 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:26.889922 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" podUID="cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0" containerName="maas-api" containerID="cri-o://958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de" gracePeriod=30 Jul 24 16:23:27.147618 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.147562 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:23:27.292113 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.292075 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf66c\" (UniqueName: \"kubernetes.io/projected/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-kube-api-access-wf66c\") pod \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " Jul 24 16:23:27.292113 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.292109 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-openshift-service-ca\") pod \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " Jul 24 16:23:27.292372 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.292211 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-maas-api-tls\") pod \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\" (UID: \"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0\") " Jul 24 16:23:27.292557 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.292534 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0" (UID: "cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:23:27.294437 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.294413 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0" (UID: "cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:23:27.294570 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.294547 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-kube-api-access-wf66c" (OuterVolumeSpecName: "kube-api-access-wf66c") pod "cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0" (UID: "cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0"). InnerVolumeSpecName "kube-api-access-wf66c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:23:27.393262 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.393228 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf66c\" (UniqueName: \"kubernetes.io/projected/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-kube-api-access-wf66c\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:23:27.393262 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.393255 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:23:27.393262 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.393265 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:23:27.429235 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.429154 2571 generic.go:356] "Generic (PLEG): container finished" podID="cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0" containerID="958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de" exitCode=0 Jul 24 16:23:27.429235 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.429207 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" Jul 24 16:23:27.429394 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.429253 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" event={"ID":"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0","Type":"ContainerDied","Data":"958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de"} Jul 24 16:23:27.429394 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.429311 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g" event={"ID":"cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0","Type":"ContainerDied","Data":"dba99d81ec64b90f06ce232491cb1e016fed690d003870b6319a2c77981c9eff"} Jul 24 16:23:27.429394 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.429334 2571 scope.go:117] "RemoveContainer" containerID="958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de" Jul 24 16:23:27.439100 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.439083 2571 scope.go:117] "RemoveContainer" containerID="958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de" Jul 24 16:23:27.439341 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:23:27.439323 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de\": container with ID starting with 958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de not found: ID does not exist" containerID="958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de" Jul 24 16:23:27.439387 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.439350 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de"} err="failed to get container status \"958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de\": rpc error: code = NotFound desc = could not find container \"958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de\": container with ID starting with 958fe5cb8340b30d81a9cbf0ad214f07441725bed101f56f55f8157f6ea844de not found: ID does not exist" Jul 24 16:23:27.451097 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.451075 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g"] Jul 24 16:23:27.457312 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:27.457290 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-ebf027-57d6b545fb-8bh2g"] Jul 24 16:23:29.250681 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:29.250649 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0" path="/var/lib/kubelet/pods/cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0/volumes" Jul 24 16:23:59.720033 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:59.719987 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf"] Jul 24 16:23:59.720651 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:59.720287 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" podUID="7b690939-7fc4-4e9e-bfb2-9a45633d2a5a" containerName="maas-api" containerID="cri-o://1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c" gracePeriod=30 Jul 24 16:23:59.967004 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:23:59.966977 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:24:00.005743 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.005600 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-maas-api-tls\") pod \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " Jul 24 16:24:00.005743 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.005642 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-openshift-service-ca\") pod \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " Jul 24 16:24:00.005743 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.005686 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf9sr\" (UniqueName: \"kubernetes.io/projected/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-kube-api-access-kf9sr\") pod \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\" (UID: \"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a\") " Jul 24 16:24:00.022192 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.006025 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7b690939-7fc4-4e9e-bfb2-9a45633d2a5a" (UID: "7b690939-7fc4-4e9e-bfb2-9a45633d2a5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:24:00.022192 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.008152 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7b690939-7fc4-4e9e-bfb2-9a45633d2a5a" (UID: "7b690939-7fc4-4e9e-bfb2-9a45633d2a5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:24:00.022192 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.008352 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-kube-api-access-kf9sr" (OuterVolumeSpecName: "kube-api-access-kf9sr") pod "7b690939-7fc4-4e9e-bfb2-9a45633d2a5a" (UID: "7b690939-7fc4-4e9e-bfb2-9a45633d2a5a"). InnerVolumeSpecName "kube-api-access-kf9sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:24:00.107311 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.107277 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf9sr\" (UniqueName: \"kubernetes.io/projected/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-kube-api-access-kf9sr\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:24:00.107311 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.107310 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:24:00.107555 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.107323 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:24:00.616796 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.616764 2571 generic.go:356] "Generic (PLEG): container finished" podID="7b690939-7fc4-4e9e-bfb2-9a45633d2a5a" containerID="1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c" exitCode=0 Jul 24 16:24:00.616968 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.616813 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" Jul 24 16:24:00.616968 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.616851 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" event={"ID":"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a","Type":"ContainerDied","Data":"1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c"} Jul 24 16:24:00.616968 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.616914 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf" event={"ID":"7b690939-7fc4-4e9e-bfb2-9a45633d2a5a","Type":"ContainerDied","Data":"45ed39d234995c857453ebafcb4bbbe897d14b2565aedf308fa244b7d44e398c"} Jul 24 16:24:00.616968 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.616937 2571 scope.go:117] "RemoveContainer" containerID="1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c" Jul 24 16:24:00.626429 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.626414 2571 scope.go:117] "RemoveContainer" containerID="1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c" Jul 24 16:24:00.626707 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:24:00.626674 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c\": container with ID starting with 1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c not found: ID does not exist" containerID="1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c" Jul 24 16:24:00.626783 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.626725 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c"} err="failed to get container status \"1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c\": rpc error: code = NotFound desc = could not find container \"1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c\": container with ID starting with 1894c7ef6554b292e29004a8c27cca88236923281ed5b0967700570976621d5c not found: ID does not exist" Jul 24 16:24:00.688868 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.688827 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf"] Jul 24 16:24:00.693536 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:00.693514 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-5d0582-7f7b7d8d6c-xtwwf"] Jul 24 16:24:01.251580 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:24:01.251544 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b690939-7fc4-4e9e-bfb2-9a45633d2a5a" path="/var/lib/kubelet/pods/7b690939-7fc4-4e9e-bfb2-9a45633d2a5a/volumes" Jul 24 16:25:06.711047 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.711012 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv"] Jul 24 16:25:06.711588 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.711236 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" podUID="dfff39d7-0128-4186-aeb7-57754dcadde7" containerName="maas-api" containerID="cri-o://fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8" gracePeriod=30 Jul 24 16:25:06.849234 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.849197 2571 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" podUID="dfff39d7-0128-4186-aeb7-57754dcadde7" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.67:8443/health\": dial tcp 10.132.0.67:8443: connect: connection refused" Jul 24 16:25:06.964340 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.964274 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:25:06.982686 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.982648 2571 generic.go:356] "Generic (PLEG): container finished" podID="dfff39d7-0128-4186-aeb7-57754dcadde7" containerID="fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8" exitCode=0 Jul 24 16:25:06.983132 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.982737 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" event={"ID":"dfff39d7-0128-4186-aeb7-57754dcadde7","Type":"ContainerDied","Data":"fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8"} Jul 24 16:25:06.983132 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.982746 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" Jul 24 16:25:06.983132 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.982808 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv" event={"ID":"dfff39d7-0128-4186-aeb7-57754dcadde7","Type":"ContainerDied","Data":"9de4e45d41839b340868f4a99dd941e4475cbea40ff0efbeaac117849b2ca59b"} Jul 24 16:25:06.983132 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.982836 2571 scope.go:117] "RemoveContainer" containerID="fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8" Jul 24 16:25:06.993531 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.993506 2571 scope.go:117] "RemoveContainer" containerID="fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8" Jul 24 16:25:06.993884 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:25:06.993860 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8\": container with ID starting with fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8 not found: ID does not exist" containerID="fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8" Jul 24 16:25:06.993983 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:06.993895 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8"} err="failed to get container status \"fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8\": rpc error: code = NotFound desc = could not find container \"fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8\": container with ID starting with fadc59f18e5c94bf0ae88cbf49298536f762b8bed4bd85a888d5078900b81db8 not found: ID does not exist" Jul 24 16:25:07.123056 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.123019 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff39d7-0128-4186-aeb7-57754dcadde7-openshift-service-ca\") pod \"dfff39d7-0128-4186-aeb7-57754dcadde7\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " Jul 24 16:25:07.123056 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.123062 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2vd9\" (UniqueName: \"kubernetes.io/projected/dfff39d7-0128-4186-aeb7-57754dcadde7-kube-api-access-h2vd9\") pod \"dfff39d7-0128-4186-aeb7-57754dcadde7\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " Jul 24 16:25:07.123323 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.123160 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls\") pod \"dfff39d7-0128-4186-aeb7-57754dcadde7\" (UID: \"dfff39d7-0128-4186-aeb7-57754dcadde7\") " Jul 24 16:25:07.123402 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.123376 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfff39d7-0128-4186-aeb7-57754dcadde7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dfff39d7-0128-4186-aeb7-57754dcadde7" (UID: "dfff39d7-0128-4186-aeb7-57754dcadde7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:25:07.125604 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.125565 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dfff39d7-0128-4186-aeb7-57754dcadde7" (UID: "dfff39d7-0128-4186-aeb7-57754dcadde7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:25:07.132883 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.132847 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfff39d7-0128-4186-aeb7-57754dcadde7-kube-api-access-h2vd9" (OuterVolumeSpecName: "kube-api-access-h2vd9") pod "dfff39d7-0128-4186-aeb7-57754dcadde7" (UID: "dfff39d7-0128-4186-aeb7-57754dcadde7"). InnerVolumeSpecName "kube-api-access-h2vd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:25:07.224551 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.224456 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dfff39d7-0128-4186-aeb7-57754dcadde7-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:25:07.224551 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.224491 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfff39d7-0128-4186-aeb7-57754dcadde7-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:25:07.224551 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.224504 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2vd9\" (UniqueName: \"kubernetes.io/projected/dfff39d7-0128-4186-aeb7-57754dcadde7-kube-api-access-h2vd9\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:25:07.302918 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.302882 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv"] Jul 24 16:25:07.305158 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:07.305137 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-07e89c-7c49777f76-x85gv"] Jul 24 16:25:09.250518 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:09.250486 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfff39d7-0128-4186-aeb7-57754dcadde7" path="/var/lib/kubelet/pods/dfff39d7-0128-4186-aeb7-57754dcadde7/volumes" Jul 24 16:25:31.434846 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.434768 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh"] Jul 24 16:25:31.435438 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.434985 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" podUID="226d1a4e-025d-402b-a562-06308ef25162" containerName="maas-api" containerID="cri-o://ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da" gracePeriod=30 Jul 24 16:25:31.690508 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.690445 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:25:31.744614 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.744576 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdbd6\" (UniqueName: \"kubernetes.io/projected/226d1a4e-025d-402b-a562-06308ef25162-kube-api-access-fdbd6\") pod \"226d1a4e-025d-402b-a562-06308ef25162\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " Jul 24 16:25:31.744842 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.744687 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226d1a4e-025d-402b-a562-06308ef25162-openshift-service-ca\") pod \"226d1a4e-025d-402b-a562-06308ef25162\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " Jul 24 16:25:31.744842 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.744734 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226d1a4e-025d-402b-a562-06308ef25162-maas-api-tls\") pod \"226d1a4e-025d-402b-a562-06308ef25162\" (UID: \"226d1a4e-025d-402b-a562-06308ef25162\") " Jul 24 16:25:31.745060 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.745037 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226d1a4e-025d-402b-a562-06308ef25162-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "226d1a4e-025d-402b-a562-06308ef25162" (UID: "226d1a4e-025d-402b-a562-06308ef25162"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:25:31.746937 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.746905 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226d1a4e-025d-402b-a562-06308ef25162-kube-api-access-fdbd6" (OuterVolumeSpecName: "kube-api-access-fdbd6") pod "226d1a4e-025d-402b-a562-06308ef25162" (UID: "226d1a4e-025d-402b-a562-06308ef25162"). InnerVolumeSpecName "kube-api-access-fdbd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:25:31.747041 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.746937 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/226d1a4e-025d-402b-a562-06308ef25162-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "226d1a4e-025d-402b-a562-06308ef25162" (UID: "226d1a4e-025d-402b-a562-06308ef25162"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:25:31.846118 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.846082 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdbd6\" (UniqueName: \"kubernetes.io/projected/226d1a4e-025d-402b-a562-06308ef25162-kube-api-access-fdbd6\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:25:31.846118 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.846112 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/226d1a4e-025d-402b-a562-06308ef25162-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:25:31.846118 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:31.846121 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/226d1a4e-025d-402b-a562-06308ef25162-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:25:32.128687 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:32.128652 2571 generic.go:356] "Generic (PLEG): container finished" podID="226d1a4e-025d-402b-a562-06308ef25162" containerID="ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da" exitCode=0 Jul 24 16:25:32.128915 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:32.128725 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" Jul 24 16:25:32.128915 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:32.128738 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" event={"ID":"226d1a4e-025d-402b-a562-06308ef25162","Type":"ContainerDied","Data":"ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da"} Jul 24 16:25:32.128915 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:32.128789 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh" event={"ID":"226d1a4e-025d-402b-a562-06308ef25162","Type":"ContainerDied","Data":"a32f79f19fcf848f433bf77f8f1b85e783cd5e3bdd90d336ff82ced6cee1510b"} Jul 24 16:25:32.128915 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:32.128812 2571 scope.go:117] "RemoveContainer" containerID="ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da" Jul 24 16:25:32.138804 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:32.138787 2571 scope.go:117] "RemoveContainer" containerID="ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da" Jul 24 16:25:32.139081 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:25:32.139062 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da\": container with ID starting with ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da not found: ID does not exist" containerID="ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da" Jul 24 16:25:32.139134 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:32.139092 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da"} err="failed to get container status \"ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da\": rpc error: code = NotFound desc = could not find container \"ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da\": container with ID starting with ad4ded5e4a955e3885c71a3a6bdead34510188d01a0e1a0b541ad79dcfd228da not found: ID does not exist" Jul 24 16:25:32.152215 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:32.152188 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh"] Jul 24 16:25:32.156154 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:32.156131 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fafc28-949d9d4c9-gd7xh"] Jul 24 16:25:33.250870 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:33.250835 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="226d1a4e-025d-402b-a562-06308ef25162" path="/var/lib/kubelet/pods/226d1a4e-025d-402b-a562-06308ef25162/volumes" Jul 24 16:25:52.740934 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:52.740902 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g"] Jul 24 16:25:52.741417 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:52.741112 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" podUID="fcdbe9e1-63e6-430e-bcdb-fda3502aaf55" containerName="maas-api" containerID="cri-o://f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564" gracePeriod=30 Jul 24 16:25:53.005509 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.005457 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:25:53.150099 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.150066 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6fzx\" (UniqueName: \"kubernetes.io/projected/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-kube-api-access-r6fzx\") pod \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " Jul 24 16:25:53.150245 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.150159 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-openshift-service-ca\") pod \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " Jul 24 16:25:53.150245 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.150176 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-maas-api-tls\") pod \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\" (UID: \"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55\") " Jul 24 16:25:53.150541 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.150514 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fcdbe9e1-63e6-430e-bcdb-fda3502aaf55" (UID: "fcdbe9e1-63e6-430e-bcdb-fda3502aaf55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:25:53.152392 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.152371 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fcdbe9e1-63e6-430e-bcdb-fda3502aaf55" (UID: "fcdbe9e1-63e6-430e-bcdb-fda3502aaf55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 16:25:53.152483 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.152429 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-kube-api-access-r6fzx" (OuterVolumeSpecName: "kube-api-access-r6fzx") pod "fcdbe9e1-63e6-430e-bcdb-fda3502aaf55" (UID: "fcdbe9e1-63e6-430e-bcdb-fda3502aaf55"). InnerVolumeSpecName "kube-api-access-r6fzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:25:53.241768 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.241734 2571 generic.go:356] "Generic (PLEG): container finished" podID="fcdbe9e1-63e6-430e-bcdb-fda3502aaf55" containerID="f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564" exitCode=0 Jul 24 16:25:53.241945 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.241778 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" event={"ID":"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55","Type":"ContainerDied","Data":"f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564"} Jul 24 16:25:53.241945 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.241785 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" Jul 24 16:25:53.241945 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.241820 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g" event={"ID":"fcdbe9e1-63e6-430e-bcdb-fda3502aaf55","Type":"ContainerDied","Data":"eaefce2375701c2c4b5e645ce7f080a038830c83ea522fe0b33fa848a906456c"} Jul 24 16:25:53.241945 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.241842 2571 scope.go:117] "RemoveContainer" containerID="f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564" Jul 24 16:25:53.252017 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.251984 2571 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-openshift-service-ca\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:25:53.252017 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.252020 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-maas-api-tls\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:25:53.252211 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.252035 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6fzx\" (UniqueName: \"kubernetes.io/projected/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55-kube-api-access-r6fzx\") on node \"ip-10-0-132-156.ec2.internal\" DevicePath \"\"" Jul 24 16:25:53.256359 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.256293 2571 scope.go:117] "RemoveContainer" containerID="f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564" Jul 24 16:25:53.256736 ip-10-0-132-156 kubenswrapper[2571]: E0724 16:25:53.256679 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564\": container with ID starting with f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564 not found: ID does not exist" containerID="f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564" Jul 24 16:25:53.256844 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.256747 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564"} err="failed to get container status \"f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564\": rpc error: code = NotFound desc = could not find container \"f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564\": container with ID starting with f12ff47c5ee6f4a72ebe9c1d7d968a4ae89c5a0c96a3e16322e050acdea7e564 not found: ID does not exist" Jul 24 16:25:53.291808 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.291781 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g"] Jul 24 16:25:53.294641 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:53.294620 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c6c91e-69447fdbc9-lcr2g"] Jul 24 16:25:55.251782 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:55.251751 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcdbe9e1-63e6-430e-bcdb-fda3502aaf55" path="/var/lib/kubelet/pods/fcdbe9e1-63e6-430e-bcdb-fda3502aaf55/volumes" Jul 24 16:25:59.470137 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:59.470108 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:25:59.475576 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:59.475555 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:25:59.477596 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:59.477572 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:25:59.483113 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:25:59.483096 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czgtt_47325823-e4e0-4aa6-80c0-fe794b86efbe/ovn-acl-logging/0.log" Jul 24 16:26:12.686005 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:12.685969 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-65d6g_7dad3b4b-7b52-4554-9897-513cbd4b394d/manager/0.log" Jul 24 16:26:12.817890 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:12.817851 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7484f4dc-fx5sc_461fa2c5-382e-4a09-99b5-0198fb86d1f8/copy-manifests/0.log" Jul 24 16:26:12.825897 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:12.825865 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7484f4dc-fx5sc_461fa2c5-382e-4a09-99b5-0198fb86d1f8/manager/0.log" Jul 24 16:26:13.474296 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:13.474264 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79b8cf8fb9-bfx54_af843a3a-82ec-40ad-8b94-e58e2b3c3e9f/maas-api/0.log" Jul 24 16:26:14.627650 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:14.627616 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn_6d439d65-e9d3-4c75-b566-da471930ff89/util/0.log" Jul 24 16:26:14.635056 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:14.635026 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn_6d439d65-e9d3-4c75-b566-da471930ff89/pull/0.log" Jul 24 16:26:14.642654 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:14.642626 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn_6d439d65-e9d3-4c75-b566-da471930ff89/extract/0.log" Jul 24 16:26:14.761211 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:14.761182 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b_9c874534-8528-4f24-9f55-c7f9adf9336f/util/0.log" Jul 24 16:26:14.768355 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:14.768330 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b_9c874534-8528-4f24-9f55-c7f9adf9336f/pull/0.log" Jul 24 16:26:14.777660 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:14.777636 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b_9c874534-8528-4f24-9f55-c7f9adf9336f/extract/0.log" Jul 24 16:26:14.898203 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:14.898153 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv_70db67de-aa2a-4207-9418-a4ac0931d96e/util/0.log" Jul 24 16:26:14.905881 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:14.905852 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv_70db67de-aa2a-4207-9418-a4ac0931d96e/pull/0.log" Jul 24 16:26:14.913101 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:14.913080 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv_70db67de-aa2a-4207-9418-a4ac0931d96e/extract/0.log" Jul 24 16:26:15.032830 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:15.032747 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb_255e6afe-1b4e-4306-8425-9ee44cd3b78a/pull/0.log" Jul 24 16:26:15.039327 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:15.039304 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb_255e6afe-1b4e-4306-8425-9ee44cd3b78a/extract/0.log" Jul 24 16:26:15.045802 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:15.045768 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb_255e6afe-1b4e-4306-8425-9ee44cd3b78a/util/0.log" Jul 24 16:26:15.415189 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:15.415106 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wn4j8_9a44395c-e5fe-4f96-82a2-725abc821320/manager/0.log" Jul 24 16:26:15.535616 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:15.535591 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dzcn7_07dc3a68-a3b7-48c2-8df7-9195a8a831ee/kuadrant-console-plugin/0.log" Jul 24 16:26:16.774192 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:16.774162 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59fb5479dc-q7nxf_a8572c57-9ea8-4200-9418-e0f93d913329/kube-auth-proxy/0.log" Jul 24 16:26:17.039281 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:17.039192 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-55mrk_eb53ba3b-7c64-4204-927d-d99794ea3c73/payload-pre-processing/0.log" Jul 24 16:26:17.307614 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:17.307536 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67b59875d5-dh89s_9165b2df-4afe-4f54-b9e4-c16f0145e83e/router/0.log" Jul 24 16:26:17.803466 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:17.803437 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-rzktz_c6642875-2374-4c1e-aea2-816cdbbd464f/main/0.log" Jul 24 16:26:18.044761 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:18.044726 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6nh9s_b3970d43-ba51-465c-8327-093a4542addc/main/0.log" Jul 24 16:26:18.164932 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:18.164853 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ndzsb_184a3192-c10e-40b2-b5a0-995440966ee0/main/0.log" Jul 24 16:26:18.282132 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:18.282103 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-qtcm4_d6e429bb-6496-40e2-9896-16c8da5003b3/main/0.log" Jul 24 16:26:27.302359 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:27.302326 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rbhv6_6a466ffd-d49f-45ea-ba0e-27bcdaf53f85/global-pull-secret-syncer/0.log" Jul 24 16:26:27.444627 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:27.444588 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mw8jp_387a88ac-aa70-4b5f-8f71-8ff684c047a8/konnectivity-agent/0.log" Jul 24 16:26:27.497231 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:27.497191 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-156.ec2.internal_fcc72598c4b952733bb79430abb4e958/haproxy/0.log" Jul 24 16:26:31.939003 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:31.938957 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn_6d439d65-e9d3-4c75-b566-da471930ff89/extract/0.log" Jul 24 16:26:31.970221 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:31.970192 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn_6d439d65-e9d3-4c75-b566-da471930ff89/util/0.log" Jul 24 16:26:31.992340 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:31.992315 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759vkxwn_6d439d65-e9d3-4c75-b566-da471930ff89/pull/0.log" Jul 24 16:26:32.035205 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.035171 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b_9c874534-8528-4f24-9f55-c7f9adf9336f/extract/0.log" Jul 24 16:26:32.060850 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.060826 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b_9c874534-8528-4f24-9f55-c7f9adf9336f/util/0.log" Jul 24 16:26:32.084324 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.084304 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0gjw9b_9c874534-8528-4f24-9f55-c7f9adf9336f/pull/0.log" Jul 24 16:26:32.134041 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.134018 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv_70db67de-aa2a-4207-9418-a4ac0931d96e/extract/0.log" Jul 24 16:26:32.160925 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.160886 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv_70db67de-aa2a-4207-9418-a4ac0931d96e/util/0.log" Jul 24 16:26:32.193990 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.193906 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7364ngv_70db67de-aa2a-4207-9418-a4ac0931d96e/pull/0.log" Jul 24 16:26:32.229276 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.229248 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb_255e6afe-1b4e-4306-8425-9ee44cd3b78a/extract/0.log" Jul 24 16:26:32.257847 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.257823 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb_255e6afe-1b4e-4306-8425-9ee44cd3b78a/util/0.log" Jul 24 16:26:32.285677 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.285650 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1cxksb_255e6afe-1b4e-4306-8425-9ee44cd3b78a/pull/0.log" Jul 24 16:26:32.897225 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.897191 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-wn4j8_9a44395c-e5fe-4f96-82a2-725abc821320/manager/0.log" Jul 24 16:26:32.944840 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:32.944809 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-dzcn7_07dc3a68-a3b7-48c2-8df7-9195a8a831ee/kuadrant-console-plugin/0.log" Jul 24 16:26:35.104979 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:35.104949 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-pqnwh_29328376-08a7-4a9d-a472-c78d6f3137fa/cluster-monitoring-operator/0.log" Jul 24 16:26:35.471364 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:35.471280 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m75qf_e6554e04-268b-415a-a050-13cd4a345939/node-exporter/0.log" Jul 24 16:26:35.504178 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:35.504155 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m75qf_e6554e04-268b-415a-a050-13cd4a345939/kube-rbac-proxy/0.log" Jul 24 16:26:35.529640 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:35.529613 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m75qf_e6554e04-268b-415a-a050-13cd4a345939/init-textfile/0.log" Jul 24 16:26:37.481314 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:37.481281 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-kv8fw_c2a1411e-9fb7-422c-8946-169e281cdaa6/networking-console-plugin/0.log" Jul 24 16:26:38.145562 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.145529 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/2.log" Jul 24 16:26:38.152430 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.152391 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-x8299_3b92d999-385e-4c9a-9f0e-6453933bacaf/console-operator/3.log" Jul 24 16:26:38.218164 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218131 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g"] Jul 24 16:26:38.218567 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218550 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="226d1a4e-025d-402b-a562-06308ef25162" containerName="maas-api" Jul 24 16:26:38.218656 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218570 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="226d1a4e-025d-402b-a562-06308ef25162" containerName="maas-api" Jul 24 16:26:38.218656 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218582 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcdbe9e1-63e6-430e-bcdb-fda3502aaf55" containerName="maas-api" Jul 24 16:26:38.218656 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218589 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcdbe9e1-63e6-430e-bcdb-fda3502aaf55" containerName="maas-api" Jul 24 16:26:38.218656 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218604 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b690939-7fc4-4e9e-bfb2-9a45633d2a5a" containerName="maas-api" Jul 24 16:26:38.218656 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218612 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b690939-7fc4-4e9e-bfb2-9a45633d2a5a" containerName="maas-api" Jul 24 16:26:38.218656 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218622 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0" containerName="maas-api" Jul 24 16:26:38.218656 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218630 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0" containerName="maas-api" Jul 24 16:26:38.218656 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218657 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfff39d7-0128-4186-aeb7-57754dcadde7" containerName="maas-api" Jul 24 16:26:38.219074 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218666 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfff39d7-0128-4186-aeb7-57754dcadde7" containerName="maas-api" Jul 24 16:26:38.219074 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218795 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcdbe9e1-63e6-430e-bcdb-fda3502aaf55" containerName="maas-api" Jul 24 16:26:38.219074 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218810 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf17559c-c56e-4fb7-b1ec-1f1b3d8b67f0" containerName="maas-api" Jul 24 16:26:38.219074 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218822 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="226d1a4e-025d-402b-a562-06308ef25162" containerName="maas-api" Jul 24 16:26:38.219074 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218829 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b690939-7fc4-4e9e-bfb2-9a45633d2a5a" containerName="maas-api" Jul 24 16:26:38.219074 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.218838 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfff39d7-0128-4186-aeb7-57754dcadde7" containerName="maas-api" Jul 24 16:26:38.221899 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.221880 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.225073 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.225052 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zsrwm\"/\"kube-root-ca.crt\"" Jul 24 16:26:38.225163 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.225094 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zsrwm\"/\"default-dockercfg-cl78s\"" Jul 24 16:26:38.226154 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.226135 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zsrwm\"/\"openshift-service-ca.crt\"" Jul 24 16:26:38.232443 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.232424 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g"] Jul 24 16:26:38.350760 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.350727 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-podres\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.350920 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.350770 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-proc\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.350920 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.350834 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-sys\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.351029 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.351000 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-lib-modules\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.351101 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.351083 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx87b\" (UniqueName: \"kubernetes.io/projected/d568a884-c270-424b-b343-e44d0b0dd48c-kube-api-access-tx87b\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.452657 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.452554 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-lib-modules\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.452657 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.452635 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx87b\" (UniqueName: \"kubernetes.io/projected/d568a884-c270-424b-b343-e44d0b0dd48c-kube-api-access-tx87b\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.452928 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.452685 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-podres\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.452928 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.452742 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-proc\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.452928 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.452784 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-lib-modules\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.452928 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.452795 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-sys\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.452928 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.452843 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-sys\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.452928 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.452842 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-podres\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.452928 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.452855 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d568a884-c270-424b-b343-e44d0b0dd48c-proc\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.462625 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.462599 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx87b\" (UniqueName: \"kubernetes.io/projected/d568a884-c270-424b-b343-e44d0b0dd48c-kube-api-access-tx87b\") pod \"perf-node-gather-daemonset-2745g\" (UID: \"d568a884-c270-424b-b343-e44d0b0dd48c\") " pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.531798 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.531763 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:38.674800 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.674573 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g"] Jul 24 16:26:38.677610 ip-10-0-132-156 kubenswrapper[2571]: W0724 16:26:38.677582 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd568a884_c270_424b_b343_e44d0b0dd48c.slice/crio-537863a80ce0cc3738a52e08fe5d71b5ec84b3311eba42e2a99a6a18637cfd48 WatchSource:0}: Error finding container 537863a80ce0cc3738a52e08fe5d71b5ec84b3311eba42e2a99a6a18637cfd48: Status 404 returned error can't find the container with id 537863a80ce0cc3738a52e08fe5d71b5ec84b3311eba42e2a99a6a18637cfd48 Jul 24 16:26:38.772581 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.772547 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85469664c6-pft48_2b695f2e-26d0-4599-aba0-b39228388794/console/0.log" Jul 24 16:26:38.830574 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:38.830546 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-9b5h5_c730a2cb-903c-486c-bd2b-d7b63013c18a/download-server/0.log" Jul 24 16:26:39.429231 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:39.429203 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-ddvlx_7364f412-e2fc-43f2-88e4-c21f7da3ab54/volume-data-source-validator/0.log" Jul 24 16:26:39.492135 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:39.492097 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" event={"ID":"d568a884-c270-424b-b343-e44d0b0dd48c","Type":"ContainerStarted","Data":"f1c7a3e2f0f21424ecfb7fe6ba13e10c4c6d82615ab585486eb86be8aaa3c836"} Jul 24 16:26:39.492311 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:39.492158 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" event={"ID":"d568a884-c270-424b-b343-e44d0b0dd48c","Type":"ContainerStarted","Data":"537863a80ce0cc3738a52e08fe5d71b5ec84b3311eba42e2a99a6a18637cfd48"} Jul 24 16:26:39.492311 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:39.492188 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:40.615713 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:40.615665 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbcnh_a726d78d-f872-44f8-ad72-1df1b141faba/dns/0.log" Jul 24 16:26:40.640594 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:40.640558 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xbcnh_a726d78d-f872-44f8-ad72-1df1b141faba/kube-rbac-proxy/0.log" Jul 24 16:26:40.694115 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:40.694081 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pl8t9_b2d7db8e-a2bf-4aad-b776-b5b77a7c3de9/dns-node-resolver/0.log" Jul 24 16:26:41.322190 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:41.322157 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b9smv_da43f1b5-a039-43c2-be38-894330b7745e/node-ca/0.log" Jul 24 16:26:42.680472 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:42.680444 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59fb5479dc-q7nxf_a8572c57-9ea8-4200-9418-e0f93d913329/kube-auth-proxy/0.log" Jul 24 16:26:42.811976 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:42.811944 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-55mrk_eb53ba3b-7c64-4204-927d-d99794ea3c73/payload-pre-processing/0.log" Jul 24 16:26:42.872160 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:42.872130 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67b59875d5-dh89s_9165b2df-4afe-4f54-b9e4-c16f0145e83e/router/0.log" Jul 24 16:26:43.734790 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:43.734751 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dhtfb_a0c12ebe-3225-4061-953c-5dc160582a12/serve-healthcheck-canary/0.log" Jul 24 16:26:44.289156 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:44.289126 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-br6wr_9fbee300-3ce7-4e7f-8c17-e56643547910/insights-operator/0.log" Jul 24 16:26:44.289841 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:44.289825 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-br6wr_9fbee300-3ce7-4e7f-8c17-e56643547910/insights-operator/1.log" Jul 24 16:26:44.493508 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:44.493480 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k8m5d_5920135f-0896-42f3-9851-5ee72589bc8c/kube-rbac-proxy/0.log" Jul 24 16:26:44.528888 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:44.528855 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k8m5d_5920135f-0896-42f3-9851-5ee72589bc8c/exporter/0.log" Jul 24 16:26:44.569207 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:44.569125 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k8m5d_5920135f-0896-42f3-9851-5ee72589bc8c/extractor/0.log" Jul 24 16:26:45.508199 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:45.508173 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" Jul 24 16:26:45.529775 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:45.529711 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zsrwm/perf-node-gather-daemonset-2745g" podStartSLOduration=7.529676142 podStartE2EDuration="7.529676142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:26:39.508598518 +0000 UTC m=+3040.741326198" watchObservedRunningTime="2026-07-24 16:26:45.529676142 +0000 UTC m=+3046.762403819" Jul 24 16:26:47.816686 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:47.816594 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-65d6g_7dad3b4b-7b52-4554-9897-513cbd4b394d/manager/0.log" Jul 24 16:26:47.849102 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:47.849069 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7484f4dc-fx5sc_461fa2c5-382e-4a09-99b5-0198fb86d1f8/manager/0.log" Jul 24 16:26:47.872295 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:47.872271 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7b7484f4dc-fx5sc_461fa2c5-382e-4a09-99b5-0198fb86d1f8/copy-manifests/0.log" Jul 24 16:26:49.438920 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:49.438894 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7db5d47bff-b9qvr_3a8bd6c4-1c68-45c3-b13e-9abb6ba0d4d8/manager/0.log" Jul 24 16:26:49.670968 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:49.670883 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59fb5479dc-q7nxf_a8572c57-9ea8-4200-9418-e0f93d913329/kube-auth-proxy/0.log" Jul 24 16:26:49.787450 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:49.787411 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-55mrk_eb53ba3b-7c64-4204-927d-d99794ea3c73/payload-pre-processing/0.log" Jul 24 16:26:49.829995 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:49.829968 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67b59875d5-dh89s_9165b2df-4afe-4f54-b9e4-c16f0145e83e/router/0.log" Jul 24 16:26:51.517878 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:51.517790 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59fb5479dc-q7nxf_a8572c57-9ea8-4200-9418-e0f93d913329/kube-auth-proxy/0.log" Jul 24 16:26:51.628577 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:51.628546 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-55mrk_eb53ba3b-7c64-4204-927d-d99794ea3c73/payload-pre-processing/0.log" Jul 24 16:26:51.669947 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:51.669912 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67b59875d5-dh89s_9165b2df-4afe-4f54-b9e4-c16f0145e83e/router/0.log" Jul 24 16:26:55.048069 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:55.048035 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx_2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a/kube-storage-version-migrator-operator/1.log" Jul 24 16:26:55.049669 ip-10-0-132-156 kubenswrapper[2571]: I0724 16:26:55.049647 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-wcclx_2044cfb4-b2d3-4b66-864e-df2d0ebc1b2a/kube-storage-version-migrator-operator/0.log"